Building human interfaces powered by AI

Berlin | November 20 - 21, 2018 Building Human Interfaces powered by AI Christian Heilmann

Building human interfaces powered by AI Chris Heilmann (@codepo8) November 2018

All resources: aka.ms/human-ai @codepo8

Let’s talk about “Artificial Intelligence” @codepo8

Artificial Intelligence @codepo8 ▪ Is nothing new – the concepts go back to the 50ies ▪ Is quite the hype and very often misattributed ▪ Is an umbrella term for a lot of math and science around repetition, pattern recognition and machine learning ▪ Got a huge boost because of availability of hardware

The machines are watching… Florian Ziegler flickr.com/photos/damndirty/41263240134

Big brother is redundant… @codepo8 ▪ Everything we do online is monitored and recorded ▪ We often don’t realise that our data is how we pay for “free” services ▪ We’re happy to use systems that record all the time in exchange for convenience ▪ Often people don’t realise just how dangerous this can be in the wrong hands.

Everything counts in large amounts @codepo8 ▪ We create a massive amount of information – actively and without our knowledge. ▪ It is tough to make that amount of information consumable again. ▪ That’s why we have computers ▪ With cloud computing, on demand processing and advances in hardware we’re faster than ever.

Leaving invisible marks… @codepo8 ▪ By using other people’s machines and infrastructure, we leave traces ▪ This allows companies to recognise us, and accumulates a usage history ▪ This leads to better results, but can leaks data ▪ We should have more transparency about what digital legacy we left behind.

Are machines friend or foe? Florian Ziegler flickr.com/photos/damndirty/40153024740/

Artificial Intelligence Myths @codepo8 ▪ AI can’t replace a thinking, creative human ▪ AI can not magically fill gaps with perfect information – it can only compare and assume ▪ AI doesn’t learn in a creative fashion. It makes no assumptions ▪ AI has no morals and ethics, but – used wrongly – it can amplify our biases

Machines can be great tools or weapons… @codepo8 ▪ Machine Learning is all about returning assumptions ▪ We don’t get any definitive truth from algorithms, we get answers to our questions ▪ AI can answer questions, but it is up to you to ask good questions – generic questions yield assumed results.

Machines can be great tools or weapons… @codepo8 ▪ Untrained and limited data leads to terrible and biased AI results ▪ It is very easy to get either wrong deductions or false positives ▪ AI is as intelligent and good as the people who apply it

About face… @codepo8 aka.ms/face-api

About face… @codepo8 ▪ Face rectangle / Landmarks ▪ Pose (pitch/roll/yaw) ▪ Smile ▪ Gender/Age ▪ Type of glasses ▪ Makeup (lips/eye) ▪ Emotion (anger, contempt, disgust, fear, happiness, neutral, sadness, surprise) ▪ Occlusion (forehead/eye/mouth) ▪ Facial hair (moustache/beard/sideburns) ▪ Attributes: Hair (invisible, bald, colour) aka.ms/face-api

Is this you? Are those also you? @codepo8 aka.ms/face-api

Is this your driver? @codepo8 youtube.com/watch?v=aEBi4OpXU4Q

Taking it too far? @codepo8 ntechlab.com

Detecting even more… @codepo8 https://apnews.com/bf75dd1c26c947b7826d270a16e2658a

Those trustworthy avatars… @codepo8 https://blog.insightdatascience.com/ generating-custom-photo-realistic-faces-using-ai-d170b1b59255

Those trustworthy avatars… @codepo8 https://blog.insightdatascience.com/ generating-custom-photo-realistic-faces-using-ai-d170b1b59255

Automated face mapping… @codepo8 https://github.com/SpiderLabs/social_mapper

Once you are known… @codepo8 https://github.com/SpiderLabs/social_mapper

Photo by Florian Ziegler flickr.com/photos/damndirty/40153024740/ AI for humans Andreas Dantz flickr.com/photos/szene/40193567250

How AI can help humans… @codepo8 aka.ms/ai-for-good

How AI can help humans… @codepo8 ▪ Automation ▪ Error prevention ▪ Data reduction / Muffling the noise ▪ Prediction based on historical data ▪ Ploughing through massive amounts of data ▪ Creating more human interfaces

How AI can help humans… @codepo8 ▪ Automation ▪ Error prevention ▪ Data reduction / Muffling the noise ▪ Prediction based on historical data ▪ Ploughing through massive amounts of data ▪ Creating more human interfaces

Humans ▪ Messy and prone to mistakes ▪ Forget things and filter them by their biases Bots and computers… ▪ Make no mistakes, other than physical fatigue ▪ Never forget, don’t judge ▪ Bored when doing repetitive tasks ▪ Great at tedious, boring tasks ▪ When bored create more errors ▪ Repeat things with minor changes on iterations till a result is met ▪ Non-optimised communication, lots of nuances and misunderstanding @codepo8 ▪ Highly optimised, non-nuanced communication.

Humans ▪ Messy and prone to mistakes ▪ Forget things and filter them by their biases Bots and computers… ▪ Make no mistakes, other than physical fatigue ▪ Never forget, don’t judge ▪ Bored when doing repetitive tasks ▪ Great at tedious, boring tasks ▪ When bored create more errors ▪ Repeat things with minor changes on iterations till a result is met ▪ Non-optimised communication, lots of nuances and misunderstanding @codepo8 ▪ Highly optimised, non-nuanced communication.

Humans Bots and computers… ▪ Messy and prone to mistakes ▪ Forget things and filter them by their biases ▪ Bored when doing repetitive tasks ▪ When bored create more errors ▪ Non-optimised communication, lots of nuances and misunderstanding @codepo8 Data Insights Patterns ▪ Make no mistakes, other than physical fatigue ▪ Never forget, don’t judge ▪ Great at tedious, boring tasks ▪ Repeat things with minor changes on iterations till a result is met ▪ Highly optimised, non-nuanced communication.

Humans and Bots/Computers @codepo8 autodraw.com

Humans and Bots/Computers @codepo8 autodraw.com

Humans and Bots/Computers @codepo8 quickdraw.withgoogle.com

Humans and Bots/Computers @codepo8 google.com/recaptcha/intro

“Learning” from lots of images @codepo8 https://github.com/jantic/DeOldify

Humans and Bots/Computers aka.ms/nvidia-fix-image

Humans and Bots/Computers aka.ms/nvidia-fix-image

Humans and Bots/Computers aka.ms/nvidia-fix-image

Intelligent, responsive systems ▪ AI services offer us lots of data to compare our users’ input with ▪ Thus our users don’t need to speak computer but be human instead ▪ We can prevent them from making mistakes ▪ We can help getting around physical barriers Google: cloud.google.com/products/machine-learning Amazon: aws.amazon.com/machine-learning @codepo8 Microsoft: azure.microsoft.com/en-us/services/cognitive-services

Our toolkit for more human interfaces Natural language processing @codepo8 Computer Vision Sentiment analysis Speech conversion and analysis Moderation

Language and Writing @codepo8 ▪ Probably the oldest task on the web was translation ▪ This moved deeper into Natural Language Processing and Language Detection ▪ Using these, we can allow for human commands and finding out tasks by analyzing texts. “How far am I from the capital of Denmark?” “Where do I find a good restaurant around here?” “Show me documents I wrote five days ago with more than 600 words”

Computer Vision @codepo8 ▪ When text wasn’t cool enough, we added images to our web media ▪ Often we forget that not everyone can see them, and we leave them without alternative text ▪ This is where machine learning steps in to help turning an image into a dataset we can work with.

Vision and image analysis… instagram: @larryandanke @codepo8

Vision and image analysis… @codepo8

Vision and image analysis… @codepo8

Vision and image analysis… @codepo8 twitter.com/mixedhunty/status/980551155297157126

Vision and image analysis… @codepo8 #vision_api

Vision and image analysis… @codepo8 aka.ms/vision-api

Vision and image analysis… @codepo8 aka.ms/vision-api

Vision and image analysis… @codepo8 aka.ms/vision-api

Vision and image analysis… @codepo8 aka.ms/vision-api

Vision and image analysis… @codepo8 aka.ms/vision-api

Sentiment analysis @codepo8 ▪ Finding out the sentiment of a text, image or video can help with a lot of things ▪ You can navigate videos by only showing the happy parts ▪ You can detect which comment should be answered first by a help desk ▪ You can predict when drivers of cars get tired

▪ Audio interfaces are all the rage. ▪ You can allow hands-free control of devices ▪ You can have an “always on” system to help you out without having to interface with it ▪ It feels natural and has a massive Sci-Fi feeling – when it works. Speech @codepo8

Speech recognition @codepo8 aka.ms/text-to-speech

Turning sentences into commands @codepo8 luis.ai aka.ms/luis-api

Text to speech @codepo8 aka.ms/text-to-speech

Conversation as an interface @codepo8 aka.ms/conversation-ui

Speaker recognition @codepo8 aka.ms/speaker-recognition

Speaker recognition @codepo8 aka.ms/speaker-recognition

Moderation @codepo8 ▪ Some things are not meant to be consumed by people ▪ Computers don’t need counselling once they saw them – people should ▪ Known illegal and terrible content can be automatically removed

With great power comes great responsibility… @codepo8

Our responsibilities.. @codepo8 ▪ AI can be an amazing help for humans ▪ It does need transparency – if you use people as data sources, they need to know what and where it goes ▪ When people get information filtered by an algorithm, it should be an opt-in ▪ People need to have a chance to dispute when an algorithm tagged or disallowed them access.

Want to go deep? ▪ The Math behind ML ▪ The ethics of AI ▪ Working with Data using Python ▪ Machine Learning Models ▪ Deep Learning Models ▪ Reinforcement Learning Models ▪ Microsoft Professional Program Certificate in Artificial Intelligence aka.ms/learn-ai 10 courses, (8-16 hours each), 10 skills @codepo8

Want to go deep? skl.sh/christian Free with trial sign-up @codepo8

Don’t forget to have fun! @codepo8

Suz Hinton @codepo8 github.com/noopkat/face-api-emoji-face

Categorising images by gesture @codepo8 http://pointerpointer.com

Find your moves @codepo8 https://experiments.withgoogle.com/move-mirror

Stay silly… @codepo8 Cassie Evans https://codepen.io/cassie-codes/pen/jKaVqo/

Help the human @codepo8 https://charliegerard.github.io/teachable-keyboard/

Artificial Intelligence @codepo8 https://charliegerard.github.io/teachable-keyboard/

Preparing the next generation @codepo8 Linda Liukas https://helloruby.com

Thanks! Chris Heilmann Christianheilmann.com Developer-evangelism.com @codepo8