A presentation at IBM Index Conference in in San Francisco, CA, USA by Tomomi ❤ Imura
Future of the Web with Conversational Interface Tomomi Imura (@girlie_mac)
tomomi imura ● ● ● ● ● ● @ advocate open web & technology code JavaScript & Node.js write & speak about tech hack useless stuff A cat lady of the InterWeb dev relations at Slack
Talking to non-human
@ Photo by Jeopardy Productions, Inc.
— Satya Nadella, Microsoft
— Sundar Pichai, Google @
@ CC BY-SA: nextdayblinds.com
Traditional Web & App Interactions @
Modern Web & Apps with Social Interactions @
Conversational User Interactions: Siri and Alexa (Voice Assistants) Alexa, how is the weather? Hey Siri @ In various form-factors
Conversational User Interactions for Kids - with Voice @ CogniToys Dino - https://cognitoys.com
Conversational User Interactions in a robot shape - with Voice @
@
Conversational User Interactions: Google Assistant (Voice & Text) @
Conversational User Interactions: Slack Bots (Text) ! @
Slack Integrations & Bots for Better Productivity
Graphic Interface to Conversational Interface
Deliver me a large margherita pizza! What kind of crust do you want? 1. Regular crust 2. Thin crust 3. Gluten free crust Thin crust ing! h c a h c @
Conversational Interface achieves: Natural user interactions with a minimal visual interface.
No UI Clutter. Less Time Spent.
Come over to my place! Where is the address of your home? It’s 325 Ruby Street. Request a ride @
Yes, get me a ride now Your driver, Sam will pick you up in 6 minutes. Look for the red Toyota Prius! @
Alexa UX (“Voice Chrome” Examples) Listening Speaking @ https://developer.amazon.com/docs/alexa-voice-service/ux-design-overview.html
Old concepts. Technology caught up with the ideas.
HAL9000: “I’m sorry Dave, I’m afraid I can’t do that” @
@
Nuh, we’re still in control over AI. (I hope) @
Conversational Interface is:
● Intuitive ● Accessible ● Productive
Messaging Platforms ● ● ● ● ● ● ● @ Slack Facebook Messenger Telegram WeChat Kik Viver LINE etc.
Messaging + Bots for More Interactive Communications
Slack bots at
Slack bots at
TacoBot by Taco Bell https://www.tacobell.com/feed/tacobot
Natural Conversation APIs ● DialogFlow (API.ai /Google) ● Wit.ai (Facebook) ● Microsoft Bot Framework ● Motion.ai @ ● ● ● ● ● ● Chatbots.io Converse AI Recast.ai Flow XO ManyChat Fluent.ai (Voice UI) etc.
Motion.ai AIaaS Artificial Intelligence as a Service DialogFlow (API.ai)
Hi, I want to book a flight! Hi Linda! Are you flying from San Francisco, as usual? Yes, from SFO. Where are you flying to? London An airline customer service bot @ Linda may not know she is talking to a bot!
@
NLP Platforms / APIs Natural Language Processing & Cognitive platforms: ● ● ● ● ● @ IBM Watson Google Cloud Natural Language API Microsoft LUIS Amazon Lex Baidu UNIT
Build Your Own Conversational Interface
…with JavaScript
The APIs are mostly accessible with JS @ { REST }
Example: API.ai + FB Messenger const app = require(‘apiai’)(CLIENT_ACCESS_TOKEN); function sendMessage(event) { API.ai Node.js SDK let sender = event.sender.id; let text = event.message.text; let ai = app.textRequest(text, { sessionId: SESSION_STRING }); ai.on(‘response’, (response) => { // Got a response. Let’s POST to Facebook Messenger }); ai.end(); } @
Example: IBM Watson + Slack @
@
Example: IBM Watson + Slack The bot workflow: 1. Read each message on a Slack channel 2. Send the message to IBM Watson for examination 3. If the likelihood of an emotion is above the given confidence threshold post the most prominent emotion @
Example: IBM Watson + Slack HTTP POST using ExpressJS app.post(‘/events’, (req, res) => { let q = req.body; if (q.type === ‘event_callback’) { Use Slack Events API to grab the text when a user post a message if(!q.event.text) return; analyzeTone(q.event); } }); @ Pass the text data to Watson to analyze
Example: IBM Watson + Slack const watson = require(‘watson-developer-cloud’); let tone_analyzer = watson.tone_analyzer({ username: process.env.WATSON_USERNAME, password: process.env.WATSON_PASSWORD, Just initializing it w/ your API credentials }); const confidencethreshold = 0.55; tone_analyzer.tone({text: text}, (err, tone) => { tone.document_tone.tone_categories.forEach((tonecategory) => { if(tonecategory.category_id === ‘emotion_tone’){ tonecategory.tones.forEach((emotion) => { Returns emotions score in 0 to 1 if(emotion.score >= confidencethreshold) { postEmotionOnSlackChannel(emotion); }}); }}); @ }); Post the result on Slack
Example: IBM Watson + Slack + Raspberry Pi (for fun) function colorEmotion(emotion) { if (emotion.tone_id === ‘anger’) { setLED(red); } else if(emotion.tone_id === ‘joy’) { setLED(yellow); } else if(emotion.tone_id === ‘fear’) { setLED(purple); } else if(emotion.tone_id === ‘disgust’) { setLED(green); } else if(emotion.tone_id === ‘sadness’) { setLED(blue); } } @ Change the LED color to match the emotion
Ack, this sucks! I want my money back! An angry customer detected. Connect the customer with a human! @
Conversational Interface with Voice in Browser?
Project: Artificial Voice Chat
Web Speech API 1 @ & 3 http://caniuse.com/#feat=speech-recognition http://caniuse.com/#feat=speech-synthesis
Web Speech API: Speech Recognition const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition; In the current Chromium, it is still prefixed const recognition = new SpeechRecognition(); @ Get an instance of the SpeechRecognition, the controller interface
Web Speech API: Speech Recognition (Cont’d) recognition.lang = ‘en-US’; recognition.interimResults = false; Some properties recognition.start(); Methods: start(), stop(), abort() recognition.addEventListener(‘result’, (e) => { let last = e.results.length - 1; let text = e.results[last][0].transcript; }); @ Events: onresult, onerror, onaudiostarted, onaudioend, etc.
Web Speech API: Speech Recognition VOICE from a user TEXT Hello! @ “hello”
Web Speech API: Speech Synthesis const synth = window.speechSynthesis; No vendor prefix const utterance = new SpeechSynthesisUtterance(); utterance.text = ‘I am a robot’; utterance.pitch = 1.5; Properties of the SpeechSynthesisUtterance interface utterance.voice = ‘Alex’; synth.speak(utterance); @ Get available voices with synth.getVoices() method
Web Speech API: Speech Synthesis Howdy TEXT to VOICE “howdy” (voice by Alex or whoever browser is) @
@
Demo on Chrome https://webspeech.herokuapp.com/
@ https://www.smashingmagazine.com/2017/08/ai-chatbot-web-speech-api-node-js/
Conversational Interface is for human.
A bots interface really is a human interface.
Clippy 1997 - 2007
@girlie_mac girliemac.com github.com/girliemac speakerdeck.com/girlie_mac @
Attribution: Open Emoji by Emoji-One (CC-BY 4.0) @
My talk about conversational interface with JavaScript
for free. You
can too.