Let’s Get Chatty with Conversational Interface in JavaScript

A presentation at Render Conf in March 2018 in Oxford, UK by Tomomi ❤ Imura

Slide 1

Slide 1

BOTS, AI, AND JAVASCRIPT Let’s Get Chatty with Conversational Interface in JavaScript #RenderConf Tomomi Imura (@girlie_mac)

Slide 2

Slide 2

a r u m i i m o m o t _ma e i l r i g @ @ girlie_mac c ● ● ● ● ● an open web & tech advocate code JavaScript / Node.js work at Slack an advisor at Code Chrysalis a cat lady of the InterWeb

Slide 3

Slide 3

Slide 4

Slide 4

“Bots are like new applications that you can converse with. ” — Satya Nadella, Microsoft

Slide 5

Slide 5

“We will evolve in computing from a mobile first to an AI first world.” — Sundar Pichai, Google @ girlie_mac

Slide 6

Slide 6

@ girlie_mac CC BY-SA: nextdayblinds.com

Slide 7

Slide 7

Traditional Web & App Interactions @ girlie_mac

Slide 8

Slide 8

Modern Web & Apps with Social Interactions @ girlie_mac

Slide 9

Slide 9

Conversational User Interactions: Siri and Alexa (Voice Assistants) Alexa, how is the weather? Hey Siri @ girlie_mac In various form-factors

Slide 10

Slide 10

Conversational User Interactions for Kids - with Voice @ girlie_mac CogniToys Dino - https://cognitoys.com

Slide 11

Slide 11

Conversational User Interactions in a robot shape - with Voice @ girlie_mac

Slide 12

Slide 12

Conversational User Interactions: Google Assistant (Voice & Text) (GIF animation) @ girlie_mac

Slide 13

Slide 13

Conversational User Interactions: Slack Bots (Text with GUI) ! @ girlie_mac

Slide 14

Slide 14

Graphic Interface to Conversational Interface

Slide 15

Slide 15

Deliver me a large margherita pizza! What kind of crust do you want? 1. Regular crust 2. Thin crust 3. Gluten free crust Thin crust ing! h c a h c @ girlie_mac

Slide 16

Slide 16

Conversational Interface achieves: Natural user interactions with a minimal visual interface.

Slide 17

Slide 17

No UI Clutter. Less Time Spent.

Slide 18

Slide 18

Come over to my place! Where is the address of your home? It’s 325 Ruby Street. Request a ride @ girlie_mac

Slide 19

Slide 19

Yes, get me a ride now Your driver, Sam will pick you up in 6 minutes. Look for the red Toyota Prius! @ girlie_mac

Slide 20

Slide 20

Alexa UX (“Voice Chrome” Examples) Listening Speaking (GIF animation) @ girlie_mac https://developer.amazon.com/docs/alexa-voice-service/ux-design-overview.html

Slide 21

Slide 21

Conversational Interface is:

Slide 22

Slide 22

● Intuitive ● Accessible ● Productive

Slide 23

Slide 23

Messaging Platforms ● ● ● ● ● ● ● Slack Facebook Messenger Telegram WeChat Kik Viver LINE etc. @ girlie_mac

Slide 24

Slide 24

Messaging + Bots for More Interactive Communications

Slide 25

Slide 25

Slack bots at

Slide 26

Slide 26

Slack bots at

Slide 27

Slide 27

TacoBot by Taco Bell https://www.tacobell.com/feed/tacobot

Slide 28

Slide 28

Natural Conversation APIs ● DialogFlow (API.ai /Google) ● Wit.ai (Facebook) ● Microsoft Bot Framework ● Motion.ai @ girlie_mac ● ● ● ● ● ● Chatbots.io Converse AI Landbot.io Recast.ai ManyChat Fluent.ai (Voice UI) etc.

Slide 29

Slide 29

@ girlie_mac

Slide 30

Slide 30

Motion.ai AIaaS Artificial Intelligence as a Service DialogFlow (API.ai)

Slide 31

Slide 31

ManyChat @ girlie_mac

Slide 32

Slide 32

Hi, I want to book a flight! Hi Linda! Are you flying from San Francisco, as usual? Yes, from SFO. Where are you flying to? London An airline customer service bot @ girlie_mac Linda may not know she is talking to a bot!

Slide 33

Slide 33

@ girlie_mac Using DialogFlow

Slide 34

Slide 34

More Powerful NLP Platforms & APIs Natural Language Processing & Cognitive platforms: ● ● ● ● ● IBM Watson Google Cloud Natural Language API Microsoft LUIS Amazon Lex Baidu UNIT @ girlie_mac

Slide 35

Slide 35

Conversational Interface With JavaScript

Slide 36

Slide 36

The APIs are mostly accessible with JS @ girlie_mac { REST } JS SDK

Slide 37

Slide 37

Example: DialogFlow (API.ai) + FB Messenger const app = require(‘apiai’)(CLIENT_ACCESS_TOKEN); function sendMessage(event) { API.ai Node.js SDK let sender = event.sender.id; let text = event.message.text; let ai = app.textRequest(text, { sessionId: SESSION_STRING }); ai.on(‘response’, (response) => { // Got a response from api.ai let aiText = response.result.fulfillment.speech; // Then POST to https://graph.facebook.com/v2.6/me/messages … }); ai.end(); @ girlie_mac }

Slide 38

Slide 38

Example: DialogFlow (API.ai) + FB Messenger @ girlie_mac

Slide 39

Slide 39

Example: Slack Message Sentiment Analysis with IBM Watson (GIF animation) @ girlie_mac

Slide 40

Slide 40

Example: IBM Watson + Slack The bot workflow: 1. Read each message on a Slack channel 2. Send the message to IBM Watson for examination 3. If the likelihood of an emotion is above the given confidence threshold post the most prominent emotion @ girlie_mac

Slide 41

Slide 41

Example: IBM Watson + Slack HTTP POST using ExpressJS app.post(‘/events’, (req, res) => { let q = req.body; if (q.type === ‘event_callback’) { Use Slack Events API to grab the text when a user post a message if(!q.event.text) return; analyzeTone(q.event); } }); @ girlie_mac Pass the text data to Watson to analyze

Slide 42

Slide 42

Example: IBM Watson + Slack const watson = require(‘watson-developer-cloud’); let tone_analyzer = watson.tone_analyzer({ username: process.env.WATSON_USERNAME, password: process.env.WATSON_PASSWORD, Just initializing it w/ your API credentials }); const confidencethreshold = 0.55; tone_analyzer.tone({text: text}, (err, tone) => { tone.document_tone.tone_categories.forEach((tonecategory) => { if(tonecategory.category_id === ‘emotion_tone’){ tonecategory.tones.forEach((emotion) => { Returns emotions score in 0 to 1 if(emotion.score >= confidencethreshold) { postEmotionOnSlackChannel(emotion); }}); }}); @ girlie_mac }); Post the result on Slack

Slide 43

Slide 43

Example: IBM Watson + Slack + Raspberry Pi (for fun) function colorEmotion(emotion) { if (emotion.tone_id === ‘anger’) { setLED(red); } else if(emotion.tone_id === ‘joy’) { setLED(yellow); } else if(emotion.tone_id === ‘fear’) { setLED(purple); } else if(emotion.tone_id === ‘disgust’) { setLED(green); } else if(emotion.tone_id === ‘sadness’) { setLED(blue); } } @ girlie_mac Change the LED color to match the emotion

Slide 44

Slide 44

@ girlie_mac

Slide 45

Slide 45

Ack, this sucks! I want my money back! An angry customer detected. Connect the customer with a human! @ girlie_mac

Slide 46

Slide 46

Conversational Interface with Voice in Browser?

Slide 47

Slide 47

Project: Artificial Voice Chat

  1. User talk to browser Voice command: Voice -> Text Hello! 1 3 Howdy
  2. Browser speaks back Text -> Synthetic Voice 2 Text
  3. Generate Artificial reply Using the 3rd party NLP API @ girlie_mac

Slide 48

Slide 48

Web Speech API 1 Speech Recognition & @ girlie_mac 3 Speech Synthesis http://caniuse.com/#feat=speech-recognition http://caniuse.com/#feat=speech-synthesis

Slide 49

Slide 49

Web Speech API: Speech Recognition const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition; In the current Chromium, it is still prefixed const recognition = new SpeechRecognition(); @ girlie_mac Get an instance of the SpeechRecognition, the controller interface

Slide 50

Slide 50

Web Speech API: Speech Recognition (Cont’d) recognition.lang = ‘en-US’; recognition.interimResults = false; Some properties recognition.start(); Methods: start(), stop(), abort() recognition.addEventListener(‘result’, (e) => { let last = e.results.length - 1; let text = e.results[last][0].transcript; }); @ girlie_mac Events: onresult, onerror, onaudiostarted, onaudioend, etc.

Slide 51

Slide 51

Web Speech API: Speech Recognition - Mic Access Permission @ girlie_mac

Slide 52

Slide 52

Web Speech API: Speech Recognition VOICE from a user TEXT Hello! “hello” Speech Recognition demo on CodePen https://codepen.io/girliemac/pen/dmpxgv @ girlie_mac

Slide 53

Slide 53

Web Speech API: Speech Synthesis const synth = window.speechSynthesis; No vendor prefix const utterance = new SpeechSynthesisUtterance(); utterance.text = ‘I am a robot’; utterance.pitch = 1.5; utterance.lang = ‘ja-JP’; synth.speak(utterance); @ girlie_mac Properties of the SpeechSynthesisUtterance interface

Slide 54

Slide 54

Web Speech API: Speech Synthesis Howdy TEXT to VOICE “howdy” (voice by Alex or whoever it is) Speech Recognition demo on CodePen https://codepen.io/girliemac/pen/dmpxgv @ girlie_mac

Slide 55

Slide 55

Project: Artificial Voice Chat @ girlie_mac

Slide 56

Slide 56

Demo time! https://webspeech.herokuapp.com/

Slide 57

Slide 57

github.com/girliemac/ web-speech-ai Article: smashingmagazine.com/ 2017/08/ai-chatbot-webspeech-api-node-js/ @ girlie_mac

Slide 58

Slide 58

Conversational Interface is for human.

Slide 59

Slide 59

A bots interface really is a human interface.

Slide 60

Slide 60

Remember me? Clippy 1997 - 2007

Slide 61

Slide 61

Thank you… Ohhhh , wait!. There ’s one more slide… @ girlie_mac

Slide 62

Slide 62

AI or cats, which will take over us first? @ girlie_mac

Slide 63

Slide 63

@girlie_mac Thank you, really! girliemac.com github.com/girliemac speakerdeck.com/girlie_mac BY-SA @ girlie_mac

Slide 64

Slide 64

Attribution: Open Emoji by Emoji-One (CC-BY 4.0) @ girlie_mac