Building Human Interfaces Powered by AI

A presentation at Web Unleashed in September 2018 in Toronto, ON, Canada by Chris Heilmann

Slide 1

Slide 1

Artificial intelligence for humans… Chris Heilmann (@codepo8) April 2018

Slide 2

Slide 2

All resources: aka.ms/human-ai @codepo8

Slide 3

Slide 3

Let’s talk about “Artificial Intelligence” @codepo8

Slide 4

Slide 4

Artificial Intelligence @codepo8 ▪ Is nothing new – the concepts go back to the 50ies ▪ Is quite the hype and very often misattributed ▪ Is an umbrella term for a lot of math and science around repetition, pattern recognition and machine learning ▪ Got a huge boost because of availability of hardware

Slide 5

Slide 5

CGP Grey: “How machines learn” @codepo8 youtube.com/watch?v=R9OHn5ZF4Uo

Slide 6

Slide 6

The machines are watching… Florian Ziegler flickr.com/photos/damndirty/41263240134

Slide 7

Slide 7

Big brother is redundant… @codepo8 ▪ Everything we do online is monitored and recorded ▪ We often don’t realise that our data is how we pay for “free” services ▪ We’re happy to use systems that record all the time in exchange for convenience ▪ Often people don’t realise just how dangerous this can be in the wrong hands.

Slide 8

Slide 8

Everything counts in large amounts @codepo8 ▪ We create a massive amount of information – actively and without our knowledge. ▪ It is tough to make that amount of information consumable again. ▪ That’s why we have computers ▪ With cloud computing, on demand processing and advances in hardware we’re faster than ever.

Slide 9

Slide 9

Leaving invisible marks… @codepo8 ▪ By using other people’s machines and infrastructure, we leave traces ▪ This allows companies to recognise us, and accumulates a usage history ▪ This leads to better results, but can leaks data ▪ We should have more transparency about what digital legacy we left behind.

Slide 10

Slide 10

Are machines friend or foe? Florian Ziegler flickr.com/photos/damndirty/40153024740/

Slide 11

Slide 11

Artificial Intelligence Myths @codepo8 ▪ AI can’t replace a thinking, creative human ▪ AI can not magically fill gaps with perfect information – it can only compare and assume ▪ AI doesn’t learn in a creative fashion. It makes no assumptions ▪ AI has no morals and ethics, but – used wrongly – it can amplify our biases

Slide 12

Slide 12

Machines can be great tools or weapons… @codepo8 ▪ Machine Learning is all about returning assumptions ▪ We don’t get any definitive truth from algorithms, we get answers to our questions ▪ AI can answer questions, but it is up to you to ask good questions – generic questions yield assumed results.

Slide 13

Slide 13

Machines can be great tools or weapons… @codepo8 ▪ Untrained and limited data leads to terrible and biased AI results ▪ It is very easy to get either wrong deductions or false positives ▪ AI is as intelligent and good as the people who apply it

Slide 14

Slide 14

About face… @codepo8 aka.ms/face-api

Slide 15

Slide 15

About face… @codepo8 ▪ Face rectangle / Landmarks ▪ Pose (pitch/roll/yaw) ▪ Smile ▪ Gender/Age ▪ Type of glasses ▪ Makeup (lips/eye) ▪ Emotion (anger, contempt, disgust, fear, happiness, neutral, sadness, surprise) ▪ Occlusion (forehead/eye/mouth) ▪ Facial hair (moustache/beard/sideburns) ▪ Attributes: Hair (invisible, bald, colour) aka.ms/face-api

Slide 16

Slide 16

Is this you? Are those also you? @codepo8 aka.ms/face-api

Slide 17

Slide 17

Is this your driver? @codepo8 youtube.com/watch?v=aEBi4OpXU4Q

Slide 18

Slide 18

Taking it too far? @codepo8 ntechlab.com

Slide 19

Slide 19

Photo by Florian Ziegler flickr.com/photos/damndirty/40153024740/ AI for humans Andreas Dantz flickr.com/photos/szene/40193567250

Slide 20

Slide 20

How AI can help humans… @codepo8 ▪ Automation ▪ Error prevention ▪ Data reduction / Muffling the noise ▪ Prediction based on historical data ▪ Ploughing through massive amounts of data ▪ Creating more human interfaces

Slide 21

Slide 21

How AI can help humans… @codepo8 aka.ms/ai-for-good

Slide 22

Slide 22

How AI can help humans… @codepo8 twitter.com/jennifermarsman/status/997655817967976448

Slide 23

Slide 23

How AI can help humans… @codepo8 ▪ Automation ▪ Error prevention ▪ Data reduction / Muffling the noise ▪ Prediction based on historical data ▪ Ploughing through massive amounts of data ▪ Creating more human interfaces

Slide 24

Slide 24

Humans Bots and computers… ▪ Messy and prone to mistakes ▪ Forget things and filter them by their biases ▪ Bored when doing repetitive tasks ▪ When bored create more errors ▪ Non-optimised communication, lots of nuances and misunderstanding @codepo8 Data Insights Patterns ▪ Make no mistakes, other than physical fatigue ▪ Never forget, don’t judge ▪ Great at tedious, boring tasks ▪ Repeat things with minor changes on iterations till a result is met ▪ Highly optimised, non-nuanced communication.

Slide 25

Slide 25

Humans and Bots/Computers @codepo8 autodraw.com

Slide 26

Slide 26

Humans and Bots/Computers @codepo8 autodraw.com

Slide 27

Slide 27

Humans and Bots/Computers @codepo8 quickdraw.withgoogle.com

Slide 28

Slide 28

Humans and Bots/Computers @codepo8 google.com/recaptcha/intro

Slide 29

Slide 29

Intelligent, responsive systems ▪ AI services offer us lots of data to compare our users’ input with ▪ Thus our users don’t need to speak computer but be human instead ▪ We can prevent them from making mistakes ▪ We can help getting around physical barriers Google: cloud.google.com/products/machine-learning Amazon: aws.amazon.com/machine-learning @codepo8 Microsoft: azure.microsoft.com/en-us/services/cognitive-services

Slide 30

Slide 30

Our toolkit for more human interfaces Natural language processing @codepo8 Computer Vision Sentiment analysis Speech conversion and analysis Moderation

Slide 31

Slide 31

Language and Writing @codepo8 ▪ Probably the oldest task on the web was translation ▪ This moved deeper into Natural Language Processing and Language Detection ▪ Using these, we can allow for human commands and finding out tasks by analyzing texts. “How far am I from the capital of Denmark?” “Where do I find a good restaurant around here?” “Show me documents I wrote five days ago with more than 600 words”

Slide 32

Slide 32

Computer Vision @codepo8 ▪ When text wasn’t cool enough, we added images to our web media ▪ Often we forget that not everyone can see them, and we leave them without alternative text ▪ This is where machine learning steps in to help turning an image into a dataset we can work with.

Slide 33

Slide 33

Vision and image analysis… instagram: @larryandanke @codepo8

Slide 34

Slide 34

Vision and image analysis… @codepo8

Slide 35

Slide 35

Vision and image analysis… @codepo8

Slide 36

Slide 36

Vision and image analysis… @codepo8 twitter.com/mixedhunty/status/980551155297157126

Slide 37

Slide 37

Vision and image analysis… @codepo8 #vision_api

Slide 38

Slide 38

Vision and image analysis… @codepo8 aka.ms/vision-api

Slide 39

Slide 39

Vision and image analysis… @codepo8 aka.ms/vision-api

Slide 40

Slide 40

Vision and image analysis… @codepo8 aka.ms/vision-api

Slide 41

Slide 41

Vision and image analysis… @codepo8 aka.ms/vision-api

Slide 42

Slide 42

Sentiment analysis @codepo8 ▪ Finding out the sentiment of a text, image or video can help with a lot of things ▪ You can navigate videos by only showing the happy parts ▪ You can detect which comment should be answered first by a help desk ▪ You can predict when drivers of cars get tired

Slide 43

Slide 43

Sentiment analysis @codepo8 github.com/noopkat/face-api-emoji-face

Slide 44

Slide 44

▪ Audio interfaces are all the rage. ▪ You can allow hands-free control of devices ▪ You can have an “always on” system to help you out without having to interface with it ▪ It feels natural and has a massive Sci-Fi feeling – when it works. Speech @codepo8

Slide 45

Slide 45

Text to speech @codepo8 aka.ms/text-to-speech

Slide 46

Slide 46

Speech recognition @codepo8 aka.ms/text-to-speech

Slide 47

Slide 47

Turning sentences into commands @codepo8 luis.ai aka.ms/luis-api

Slide 48

Slide 48

Conversation as an interface @codepo8 aka.ms/conversation-ui

Slide 49

Slide 49

Speaker recognition @codepo8 aka.ms/speaker-recognition

Slide 50

Slide 50

Speaker recognition @codepo8 aka.ms/speaker-recognition

Slide 51

Slide 51

Moderation @codepo8 ▪ Some things are not meant to be consumed by people ▪ Computers don’t need counselling once they saw them – people should ▪ Known illegal and terrible content can be automatically removed

Slide 52

Slide 52

With great power comes great responsibility… @codepo8

Slide 53

Slide 53

Our responsibilities.. @codepo8 ▪ AI can be an amazing help for humans ▪ It does need transparency – if you use people as data sources, they need to know what and where it goes ▪ When people get information filtered by an algorithm, it should be an opt-in ▪ People need to have a chance to dispute when an algorithm tagged or disallowed them access.

Slide 54

Slide 54

Want to go deep? ▪ The Math behind ML ▪ The ethics of AI ▪ Working with Data using Python ▪ Machine Learning Models ▪ Deep Learning Models ▪ Reinforcement Learning Models ▪ Microsoft Professional Program Certificate in Artificial Intelligence aka.ms/learn-ai 10 courses, (8-16 hours each), 10 skills @codepo8

Slide 55

Slide 55

Want to go deep? skl.sh/christian Free with trial sign-up @codepo8

Slide 56

Slide 56

Thanks! Chris Heilmann Christianheilmann.com Developer-evangelism.com @codepo8