Let’s Get Chatty with Conversational Interface in JavaScript

A presentation at Render Conf in March 2018 in Oxford, UK by Tomomi ❤ Imura

BOTS, AI, AND JAVASCRIPT Let’s Get Chatty with Conversational Interface in JavaScript #RenderConf Tomomi Imura (@girlie_mac)

a r u m i i m o m o t _ma e i l r i g @ @ girlie_mac c ● ● ● ● ● an open web & tech advocate code JavaScript / Node.js work at Slack an advisor at Code Chrysalis a cat lady of the InterWeb

“Bots are like new applications that you can converse with. ” — Satya Nadella, Microsoft

“We will evolve in computing from a mobile first to an AI first world.” — Sundar Pichai, Google @ girlie_mac

@ girlie_mac CC BY-SA: nextdayblinds.com

Traditional Web & App Interactions @ girlie_mac

Modern Web & Apps with Social Interactions @ girlie_mac

Conversational User Interactions: Siri and Alexa (Voice Assistants) Alexa, how is the weather? Hey Siri @ girlie_mac In various form-factors

Conversational User Interactions for Kids - with Voice @ girlie_mac CogniToys Dino - https://cognitoys.com

Conversational User Interactions in a robot shape - with Voice @ girlie_mac

Conversational User Interactions: Google Assistant (Voice & Text) (GIF animation) @ girlie_mac

Conversational User Interactions: Slack Bots (Text with GUI) ! @ girlie_mac

Graphic Interface to Conversational Interface

Deliver me a large margherita pizza! What kind of crust do you want? 1. Regular crust 2. Thin crust 3. Gluten free crust Thin crust ing! h c a h c @ girlie_mac

Conversational Interface achieves: Natural user interactions with a minimal visual interface.

No UI Clutter. Less Time Spent.

Come over to my place! Where is the address of your home? It’s 325 Ruby Street. Request a ride @ girlie_mac

Yes, get me a ride now Your driver, Sam will pick you up in 6 minutes. Look for the red Toyota Prius! @ girlie_mac

Alexa UX (“Voice Chrome” Examples) Listening Speaking (GIF animation) @ girlie_mac https://developer.amazon.com/docs/alexa-voice-service/ux-design-overview.html

Conversational Interface is:

● Intuitive ● Accessible ● Productive

Messaging Platforms ● ● ● ● ● ● ● Slack Facebook Messenger Telegram WeChat Kik Viver LINE etc. @ girlie_mac

Messaging + Bots for More Interactive Communications

Slack bots at

Slack bots at

TacoBot by Taco Bell https://www.tacobell.com/feed/tacobot

Natural Conversation APIs ● DialogFlow (API.ai /Google) ● Wit.ai (Facebook) ● Microsoft Bot Framework ● Motion.ai @ girlie_mac ● ● ● ● ● ● Chatbots.io Converse AI Landbot.io Recast.ai ManyChat Fluent.ai (Voice UI) etc.

@ girlie_mac

Motion.ai AIaaS Artificial Intelligence as a Service DialogFlow (API.ai)

ManyChat @ girlie_mac

Hi, I want to book a flight! Hi Linda! Are you flying from San Francisco, as usual? Yes, from SFO. Where are you flying to? London An airline customer service bot @ girlie_mac Linda may not know she is talking to a bot!

@ girlie_mac Using DialogFlow

More Powerful NLP Platforms & APIs Natural Language Processing & Cognitive platforms: ● ● ● ● ● IBM Watson Google Cloud Natural Language API Microsoft LUIS Amazon Lex Baidu UNIT @ girlie_mac

Conversational Interface With JavaScript

The APIs are mostly accessible with JS @ girlie_mac { REST } JS SDK

Example: DialogFlow (API.ai) + FB Messenger const app = require(‘apiai’)(CLIENT_ACCESS_TOKEN); function sendMessage(event) { API.ai Node.js SDK let sender = event.sender.id; let text = event.message.text; let ai = app.textRequest(text, { sessionId: SESSION_STRING }); ai.on(‘response’, (response) => { // Got a response from api.ai let aiText = response.result.fulfillment.speech; // Then POST to https://graph.facebook.com/v2.6/me/messages … }); ai.end(); @ girlie_mac }

Example: DialogFlow (API.ai) + FB Messenger @ girlie_mac

Example: Slack Message Sentiment Analysis with IBM Watson (GIF animation) @ girlie_mac

Example: IBM Watson + Slack The bot workflow: 1. Read each message on a Slack channel 2. Send the message to IBM Watson for examination 3. If the likelihood of an emotion is above the given confidence threshold post the most prominent emotion @ girlie_mac

Example: IBM Watson + Slack HTTP POST using ExpressJS app.post(‘/events’, (req, res) => { let q = req.body; if (q.type === ‘event_callback’) { Use Slack Events API to grab the text when a user post a message if(!q.event.text) return; analyzeTone(q.event); } }); @ girlie_mac Pass the text data to Watson to analyze

Example: IBM Watson + Slack const watson = require(‘watson-developer-cloud’); let tone_analyzer = watson.tone_analyzer({ username: process.env.WATSON_USERNAME, password: process.env.WATSON_PASSWORD, Just initializing it w/ your API credentials }); const confidencethreshold = 0.55; tone_analyzer.tone({text: text}, (err, tone) => { tone.document_tone.tone_categories.forEach((tonecategory) => { if(tonecategory.category_id === ‘emotion_tone’){ tonecategory.tones.forEach((emotion) => { Returns emotions score in 0 to 1 if(emotion.score >= confidencethreshold) { postEmotionOnSlackChannel(emotion); }}); }}); @ girlie_mac }); Post the result on Slack

Example: IBM Watson + Slack + Raspberry Pi (for fun) function colorEmotion(emotion) { if (emotion.tone_id === ‘anger’) { setLED(red); } else if(emotion.tone_id === ‘joy’) { setLED(yellow); } else if(emotion.tone_id === ‘fear’) { setLED(purple); } else if(emotion.tone_id === ‘disgust’) { setLED(green); } else if(emotion.tone_id === ‘sadness’) { setLED(blue); } } @ girlie_mac Change the LED color to match the emotion

@ girlie_mac

Ack, this sucks! I want my money back! An angry customer detected. Connect the customer with a human! @ girlie_mac

Conversational Interface with Voice in Browser?

Project: Artificial Voice Chat
User talk to browser Voice command: Voice -> Text Hello! 1 3 Howdy
Browser speaks back Text -> Synthetic Voice 2 Text
Generate Artificial reply Using the 3rd party NLP API @ girlie_mac

Web Speech API 1 Speech Recognition & @ girlie_mac 3 Speech Synthesis http://caniuse.com/#feat=speech-recognition http://caniuse.com/#feat=speech-synthesis

Web Speech API: Speech Recognition const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition; In the current Chromium, it is still prefixed const recognition = new SpeechRecognition(); @ girlie_mac Get an instance of the SpeechRecognition, the controller interface

Web Speech API: Speech Recognition (Cont’d) recognition.lang = ‘en-US’; recognition.interimResults = false; Some properties recognition.start(); Methods: start(), stop(), abort() recognition.addEventListener(‘result’, (e) => { let last = e.results.length - 1; let text = e.results[last][0].transcript; }); @ girlie_mac Events: onresult, onerror, onaudiostarted, onaudioend, etc.

Web Speech API: Speech Recognition - Mic Access Permission @ girlie_mac

Web Speech API: Speech Recognition VOICE from a user TEXT Hello! “hello” Speech Recognition demo on CodePen https://codepen.io/girliemac/pen/dmpxgv @ girlie_mac

Web Speech API: Speech Synthesis const synth = window.speechSynthesis; No vendor prefix const utterance = new SpeechSynthesisUtterance(); utterance.text = ‘I am a robot’; utterance.pitch = 1.5; utterance.lang = ‘ja-JP’; synth.speak(utterance); @ girlie_mac Properties of the SpeechSynthesisUtterance interface

Web Speech API: Speech Synthesis Howdy TEXT to VOICE “howdy” (voice by Alex or whoever it is) Speech Recognition demo on CodePen https://codepen.io/girliemac/pen/dmpxgv @ girlie_mac

Project: Artificial Voice Chat @ girlie_mac

Demo time! https://webspeech.herokuapp.com/

github.com/girliemac/ web-speech-ai Article: smashingmagazine.com/ 2017/08/ai-chatbot-webspeech-api-node-js/ @ girlie_mac

Conversational Interface is for human.

A bots interface really is a human interface.

Remember me? Clippy 1997 - 2007

Thank you… Ohhhh , wait!. There ’s one more slide… @ girlie_mac

AI or cats, which will take over us first? @ girlie_mac

@girlie_mac Thank you, really! girliemac.com github.com/girliemac speakerdeck.com/girlie_mac BY-SA @ girlie_mac

Attribution: Open Emoji by Emoji-One (CC-BY 4.0) @ girlie_mac

Tomomi ❤ Imura
@girliemac

1 / 64

My talk at RenderConf in Oxford, UK