Future of the Web with Conversational Interface (in JavaScript!)

Slide 1

Future of the Web with Conversational Interface Tomomi Imura (@girlie_mac)

Slide 2

tomomi imura ● ● ● ● ● ● @ advocate open web & technology code JavaScript & Node.js write & speak about tech hack useless stuff A cat lady of the InterWeb dev relations at Slack

Slide 3

Talking to non-human

Slide 4

@ Photo by Jeopardy Productions, Inc.

Slide 5

— Satya Nadella, Microsoft

Slide 6

— Sundar Pichai, Google @

Slide 7

@ CC BY-SA: nextdayblinds.com

Slide 8

Traditional Web & App Interactions @

Slide 9

Modern Web & Apps with Social Interactions @

Slide 10

Conversational User Interactions: Siri and Alexa (Voice Assistants) Alexa, how is the weather? Hey Siri @ In various form-factors

Slide 11

Conversational User Interactions for Kids - with Voice @ CogniToys Dino - https://cognitoys.com

Slide 12

Conversational User Interactions in a robot shape - with Voice @

Slide 13

Slide 14

Conversational User Interactions: Google Assistant (Voice & Text) @

Slide 15

Conversational User Interactions: Slack Bots (Text) ! @

Slide 16

Slack Integrations & Bots for Better Productivity

Slide 17

Graphic Interface to Conversational Interface

Slide 18

Deliver me a large margherita pizza! What kind of crust do you want? 1. Regular crust 2. Thin crust 3. Gluten free crust Thin crust ing! h c a h c @

Slide 19

Conversational Interface achieves: Natural user interactions with a minimal visual interface.

Slide 20

No UI Clutter. Less Time Spent.

Slide 21

Come over to my place! Where is the address of your home? It’s 325 Ruby Street. Request a ride @

Slide 22

Yes, get me a ride now Your driver, Sam will pick you up in 6 minutes. Look for the red Toyota Prius! @

Slide 23

Alexa UX (“Voice Chrome” Examples) Listening Speaking @ https://developer.amazon.com/docs/alexa-voice-service/ux-design-overview.html

Slide 24

Old concepts. Technology caught up with the ideas.

Slide 25

HAL9000: “I’m sorry Dave, I’m afraid I can’t do that” @

Slide 26

Slide 27

Nuh, we’re still in control over AI. (I hope) @

Slide 28

Conversational Interface is:

Slide 29

● Intuitive ● Accessible ● Productive

Slide 30

Messaging Platforms ● ● ● ● ● ● ● @ Slack Facebook Messenger Telegram WeChat Kik Viver LINE etc.

Slide 31

Messaging + Bots for More Interactive Communications

Slide 32

Slack bots at

Slide 33

Slack bots at

Slide 34

TacoBot by Taco Bell https://www.tacobell.com/feed/tacobot

Slide 35

Natural Conversation APIs ● DialogFlow (API.ai /Google) ● Wit.ai (Facebook) ● Microsoft Bot Framework ● Motion.ai @ ● ● ● ● ● ● Chatbots.io Converse AI Recast.ai Flow XO ManyChat Fluent.ai (Voice UI) etc.

Slide 36

Motion.ai AIaaS Artificial Intelligence as a Service DialogFlow (API.ai)

Slide 37

Hi, I want to book a flight! Hi Linda! Are you flying from San Francisco, as usual? Yes, from SFO. Where are you flying to? London An airline customer service bot @ Linda may not know she is talking to a bot!

Slide 38

Slide 39

NLP Platforms / APIs Natural Language Processing & Cognitive platforms: ● ● ● ● ● @ IBM Watson Google Cloud Natural Language API Microsoft LUIS Amazon Lex Baidu UNIT

Slide 40

Build Your Own Conversational Interface

Slide 41

…with JavaScript

Slide 42

The APIs are mostly accessible with JS @ { REST }

Slide 43

Example: API.ai + FB Messenger const app = require(‘apiai’)(CLIENT_ACCESS_TOKEN); function sendMessage(event) { API.ai Node.js SDK let sender = event.sender.id; let text = event.message.text; let ai = app.textRequest(text, { sessionId: SESSION_STRING }); ai.on(‘response’, (response) => { // Got a response. Let’s POST to Facebook Messenger }); ai.end(); } @

Slide 44

Example: IBM Watson + Slack @

Slide 45

Slide 46

Example: IBM Watson + Slack The bot workflow: 1. Read each message on a Slack channel 2. Send the message to IBM Watson for examination 3. If the likelihood of an emotion is above the given confidence threshold post the most prominent emotion @

Slide 47

Example: IBM Watson + Slack HTTP POST using ExpressJS app.post(‘/events’, (req, res) => { let q = req.body; if (q.type === ‘event_callback’) { Use Slack Events API to grab the text when a user post a message if(!q.event.text) return; analyzeTone(q.event); } }); @ Pass the text data to Watson to analyze

Slide 48

Example: IBM Watson + Slack const watson = require(‘watson-developer-cloud’); let tone_analyzer = watson.tone_analyzer({ username: process.env.WATSON_USERNAME, password: process.env.WATSON_PASSWORD, Just initializing it w/ your API credentials }); const confidencethreshold = 0.55; tone_analyzer.tone({text: text}, (err, tone) => { tone.document_tone.tone_categories.forEach((tonecategory) => { if(tonecategory.category_id === ‘emotion_tone’){ tonecategory.tones.forEach((emotion) => { Returns emotions score in 0 to 1 if(emotion.score >= confidencethreshold) { postEmotionOnSlackChannel(emotion); }}); }}); @ }); Post the result on Slack

Slide 49

Example: IBM Watson + Slack + Raspberry Pi (for fun) function colorEmotion(emotion) { if (emotion.tone_id === ‘anger’) { setLED(red); } else if(emotion.tone_id === ‘joy’) { setLED(yellow); } else if(emotion.tone_id === ‘fear’) { setLED(purple); } else if(emotion.tone_id === ‘disgust’) { setLED(green); } else if(emotion.tone_id === ‘sadness’) { setLED(blue); } } @ Change the LED color to match the emotion

Slide 50

Ack, this sucks! I want my money back! An angry customer detected. Connect the customer with a human! @

Slide 51

Conversational Interface with Voice in Browser?

Slide 52

Project: Artificial Voice Chat

User talk to browser Voice command: Voice -> Text Hello! 1 3 Howdy
Browser speaks back Text -> Synthetic Voice 2 Text
Generate Artificial reply Using the 3rd party API @

Slide 53

Web Speech API 1 @ & 3 http://caniuse.com/#feat=speech-recognition http://caniuse.com/#feat=speech-synthesis

Slide 54

Web Speech API: Speech Recognition const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition; In the current Chromium, it is still prefixed const recognition = new SpeechRecognition(); @ Get an instance of the SpeechRecognition, the controller interface

Slide 55

Web Speech API: Speech Recognition (Cont’d) recognition.lang = ‘en-US’; recognition.interimResults = false; Some properties recognition.start(); Methods: start(), stop(), abort() recognition.addEventListener(‘result’, (e) => { let last = e.results.length - 1; let text = e.results[last][0].transcript; }); @ Events: onresult, onerror, onaudiostarted, onaudioend, etc.

Slide 56

Web Speech API: Speech Recognition VOICE from a user TEXT Hello! @ “hello”

Slide 57

Web Speech API: Speech Synthesis const synth = window.speechSynthesis; No vendor prefix const utterance = new SpeechSynthesisUtterance(); utterance.text = ‘I am a robot’; utterance.pitch = 1.5; Properties of the SpeechSynthesisUtterance interface utterance.voice = ‘Alex’; synth.speak(utterance); @ Get available voices with synth.getVoices() method