Future of the Web with Conversational Interface (in JavaScript!)

Future of the Web with Conversational Interface Tomomi Imura (@girlie_mac)

tomomi imura ● ● ● ● ● ● @ advocate open web & technology code JavaScript & Node.js write & speak about tech hack useless stuff A cat lady of the InterWeb dev relations at Slack

Talking to non-human

@ Photo by Jeopardy Productions, Inc.

— Satya Nadella, Microsoft

— Sundar Pichai, Google @

@ CC BY-SA: nextdayblinds.com

Traditional Web & App Interactions @

Modern Web & Apps with Social Interactions @

Conversational User Interactions: Siri and Alexa (Voice Assistants) Alexa, how is the weather? Hey Siri @ In various form-factors

Conversational User Interactions for Kids - with Voice @ CogniToys Dino - https://cognitoys.com

Conversational User Interactions in a robot shape - with Voice @

@

Conversational User Interactions: Google Assistant (Voice & Text) @

Conversational User Interactions: Slack Bots (Text) ! @

Slack Integrations & Bots for Better Productivity

Graphic Interface to Conversational Interface

Deliver me a large margherita pizza! What kind of crust do you want? 1. Regular crust 2. Thin crust 3. Gluten free crust Thin crust ing! h c a h c @

Conversational Interface achieves: Natural user interactions with a minimal visual interface.

No UI Clutter. Less Time Spent.

Come over to my place! Where is the address of your home? It’s 325 Ruby Street. Request a ride @

Yes, get me a ride now Your driver, Sam will pick you up in 6 minutes. Look for the red Toyota Prius! @

Alexa UX (“Voice Chrome” Examples) Listening Speaking @ https://developer.amazon.com/docs/alexa-voice-service/ux-design-overview.html

Old concepts. Technology caught up with the ideas.

HAL9000: “I’m sorry Dave, I’m afraid I can’t do that” @

@

Nuh, we’re still in control over AI. (I hope) @

Conversational Interface is:

● Intuitive ● Accessible ● Productive

Messaging Platforms ● ● ● ● ● ● ● @ Slack Facebook Messenger Telegram WeChat Kik Viver LINE etc.

Messaging + Bots for More Interactive Communications

Slack bots at

Slack bots at

TacoBot by Taco Bell https://www.tacobell.com/feed/tacobot

Natural Conversation APIs ● DialogFlow (API.ai /Google) ● Wit.ai (Facebook) ● Microsoft Bot Framework ● Motion.ai @ ● ● ● ● ● ● Chatbots.io Converse AI Recast.ai Flow XO ManyChat Fluent.ai (Voice UI) etc.

Motion.ai AIaaS Artificial Intelligence as a Service DialogFlow (API.ai)

Hi, I want to book a flight! Hi Linda! Are you flying from San Francisco, as usual? Yes, from SFO. Where are you flying to? London An airline customer service bot @ Linda may not know she is talking to a bot!

@

NLP Platforms / APIs Natural Language Processing & Cognitive platforms: ● ● ● ● ● @ IBM Watson Google Cloud Natural Language API Microsoft LUIS Amazon Lex Baidu UNIT

Build Your Own Conversational Interface

…with JavaScript

The APIs are mostly accessible with JS @ { REST }

Example: API.ai + FB Messenger const app = require(‘apiai’)(CLIENT_ACCESS_TOKEN); function sendMessage(event) { API.ai Node.js SDK let sender = event.sender.id; let text = event.message.text; let ai = app.textRequest(text, { sessionId: SESSION_STRING }); ai.on(‘response’, (response) => { // Got a response. Let’s POST to Facebook Messenger }); ai.end(); } @

Example: IBM Watson + Slack @

@

Example: IBM Watson + Slack The bot workflow: 1. Read each message on a Slack channel 2. Send the message to IBM Watson for examination 3. If the likelihood of an emotion is above the given confidence threshold post the most prominent emotion @

Example: IBM Watson + Slack HTTP POST using ExpressJS app.post(‘/events’, (req, res) => { let q = req.body; if (q.type === ‘event_callback’) { Use Slack Events API to grab the text when a user post a message if(!q.event.text) return; analyzeTone(q.event); } }); @ Pass the text data to Watson to analyze

Example: IBM Watson + Slack const watson = require(‘watson-developer-cloud’); let tone_analyzer = watson.tone_analyzer({ username: process.env.WATSON_USERNAME, password: process.env.WATSON_PASSWORD, Just initializing it w/ your API credentials }); const confidencethreshold = 0.55; tone_analyzer.tone({text: text}, (err, tone) => { tone.document_tone.tone_categories.forEach((tonecategory) => { if(tonecategory.category_id === ‘emotion_tone’){ tonecategory.tones.forEach((emotion) => { Returns emotions score in 0 to 1 if(emotion.score >= confidencethreshold) { postEmotionOnSlackChannel(emotion); }}); }}); @ }); Post the result on Slack

Example: IBM Watson + Slack + Raspberry Pi (for fun) function colorEmotion(emotion) { if (emotion.tone_id === ‘anger’) { setLED(red); } else if(emotion.tone_id === ‘joy’) { setLED(yellow); } else if(emotion.tone_id === ‘fear’) { setLED(purple); } else if(emotion.tone_id === ‘disgust’) { setLED(green); } else if(emotion.tone_id === ‘sadness’) { setLED(blue); } } @ Change the LED color to match the emotion

Ack, this sucks! I want my money back! An angry customer detected. Connect the customer with a human! @

Conversational Interface with Voice in Browser?

Project: Artificial Voice Chat

User talk to browser Voice command: Voice -> Text Hello! 1 3 Howdy
Browser speaks back Text -> Synthetic Voice 2 Text
Generate Artificial reply Using the 3rd party API @

Web Speech API 1 @ & 3 http://caniuse.com/#feat=speech-recognition http://caniuse.com/#feat=speech-synthesis

Web Speech API: Speech Recognition const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition; In the current Chromium, it is still prefixed const recognition = new SpeechRecognition(); @ Get an instance of the SpeechRecognition, the controller interface

Web Speech API: Speech Recognition (Cont’d) recognition.lang = ‘en-US’; recognition.interimResults = false; Some properties recognition.start(); Methods: start(), stop(), abort() recognition.addEventListener(‘result’, (e) => { let last = e.results.length - 1; let text = e.results[last][0].transcript; }); @ Events: onresult, onerror, onaudiostarted, onaudioend, etc.

Web Speech API: Speech Recognition VOICE from a user TEXT Hello! @ “hello”

Web Speech API: Speech Synthesis const synth = window.speechSynthesis; No vendor prefix const utterance = new SpeechSynthesisUtterance(); utterance.text = ‘I am a robot’; utterance.pitch = 1.5; Properties of the SpeechSynthesisUtterance interface utterance.voice = ‘Alex’; synth.speak(utterance); @ Get available voices with synth.getVoices() method

Web Speech API: Speech Synthesis Howdy TEXT to VOICE “howdy” (voice by Alex or whoever browser is) @

@

Demo on Chrome https://webspeech.herokuapp.com/

@ https://www.smashingmagazine.com/2017/08/ai-chatbot-web-speech-api-node-js/

Conversational Interface is for human.

A bots interface really is a human interface.

Clippy 1997 - 2007

@girlie_mac girliemac.com github.com/girliemac speakerdeck.com/girlie_mac @

Attribution: Open Emoji by Emoji-One (CC-BY 4.0) @

Future of the Web with Conversational Interface Tomomi Imura (@girlie_mac)