Creating Human interfaces with AI

Welcome to Azure Saturday 2019 Munich #AzureSaturday 18.05.2019 – Microsoft Munich – azuresaturday.de — @azuresaturday

#AzureSaturday 18.05.2019 – Microsoft Munich – azuresaturday.de – @azuresaturday Building human interfaces powered by AI Speaker: Chris Heilmann

Building human interfaces powered by AI Chris Heilmann (@codepo8) November 2018

All resources: aka.ms/human-ai @codepo8

Let’s talk about “Artificial Intelligence” @codepo8

What is the difference between Machine Learning and Artificial Intelligence? @codepo8

Machine Learning is written in Python, JavaScript… Artificial Intelligence is written in PowerPoint. @codepo8

Artificial Intelligence @codepo8 ▪ Is nothing new – the concepts go back to the 50ies ▪ Is quite the hype and very often misattributed ▪ Is an umbrella term for a lot of math and science around repetition, pattern recognition and machine learning ▪ Got a huge boost because of availability of hardware ▪ Became much more feasible because of the availability of lots of data

Reminders of “genie in the bottle” @codepo8 ▪ Fulfills our wishes seemingly with invisible magic ▪ Useful, and feels too good to be true ▪ Once released, may have a dark, sinister edge to it ▪ Hard to put back into the bottle.

Let’s start with some predictions. @codepo8 ▪ AI is the number one growth market in IT – the others are cloud and security ▪ Machine Learning is already replacing thousands of jobs – boring, terrible jobs humans should not do ▪ This is also happening in IT – we are not invincible because we know hot to exit Vim

Let’s start with some predictions. @codepo8 ▪ There is no stopping this – it is just too convenient ▪ The amount of data we create (actively or by triggering sensors) demands machines to whittle it down for us to make it consumable by humans ▪ If we as developers and decision makers in IT don’t take ownership and lead with good, ethical examples, we’ll throw away decades of work democratising computing

The machines are watching… Florian Ziegler flickr.com/photos/damndirty/41263240134

Social Credit System @codepo8 https://futurism.com/china-social-credit-system-rate-human-value/

Big brother is redundant… @codepo8 ▪ Everything we do online is monitored and recorded ▪ We often don’t realise that our data is how we pay for “free” services ▪ We’re happy to use systems that record all the time in exchange for convenience ▪ Often people don’t realise just how dangerous this can be in the wrong hands.

Everything counts in large amounts @codepo8 ▪ We create a massive amount of information – actively and without our knowledge. ▪ It is tough to make that amount of information consumable again. ▪ That’s why we have computers ▪ With cloud computing, on demand processing and advances in hardware we’re faster than ever.

Leaving invisible marks… @codepo8 ▪ By using other people’s machines and infrastructure, we leave traces ▪ This allows companies to recognise us, and accumulates a usage history ▪ This leads to better results, but can leak data ▪ We should have more transparency about what digital legacy we left behind.

Are machines friend or foe? Florian Ziegler flickr.com/photos/damndirty/40153024740/

Artificial Intelligence Myths @codepo8 ▪ AI can’t replace a thinking, creative human ▪ AI can not magically fill gaps with perfect information – it can only compare and assume ▪ AI doesn’t learn in a creative fashion. It makes no assumptions ▪ AI has no morals and ethics, but – used wrongly – it can amplify our biases

Machines can be great tools or weapons… @codepo8 ▪ Machine Learning is all about returning assumptions ▪ We don’t get any definitive truth from algorithms, we get answers to our questions ▪ AI can answer questions, but it is up to you to ask good questions – generic questions yield assumed results.

Unguided or supervised AI… @codepo8 http://inspirobot.me

It can be demanding @codepo8 http://inspirobot.me

It can mix up needs… @codepo8 http://inspirobot.me

It can be overly excited… @codepo8 http://inspirobot.me

It can be a good warning… @codepo8 http://inspirobot.me

It can be painfully humbling… @codepo8 http://inspirobot.me

Prophetic, even? @codepo8 http://inspirobot.me

Passive aggressive towards humans… @codepo8

It can be adoringly cute… @codepo8 https://twitter.com/eron_gj/status/967672260147470336

Whilst being actually kick-ass @codepo8 https://www.youtube.com/watch?v=gn4nRCC9TwQ

Machines can be great tools or weapons… @codepo8 ▪ Untrained and limited data leads to terrible and biased AI results ▪ It is very easy to get either wrong deductions or false positives ▪ AI is as intelligent and good as the people who apply it

Machine learning helps us in a few ways… @codepo8 ▪ Recommendation ▪ Prediction ▪ Classification ▪ Clustering ▪ Generation

Machines ploughing through lots of data for you. Recommendation @codepo8 ▪ “I feel lucky” moments ▪ Slack finding people in your organization ▪ Intelligent inboxes ▪ Automated photo optimization ▪ Automated tagging and alternative text: “Image may contain”

You’re doing this – you probably want this as the next thing Prediction @codepo8 ▪ Text autocompletion ▪ Task offerings ▪ Image tooling – adding photos to a collage ▪ Creating albums ▪ Offering similar music and videos ▪ Offering products that match

Sort things by what humans told you what they are and scale it up Classification @codepo8 ▪ Google surveys offering the right form elements for a question ▪ Detecting faces and asking for more information ▪ Finding anomalies in health scans and doing the same for all the ones in the system

Find own patterns and collate them Clustering @codepo8 ▪ Photo tagging and ordering ▪ Document analysis ▪ Comment filtering and triaging ▪ Video optimisation dependent on content.

Allow the machine to create things Generation @codepo8 ▪ Art style matching ▪ Generated articles from fact collection ▪ Synthesised music ▪ Filling content with tagged information (grass, houses, brick, etc…) ▪ React to human input

We need to find our place on the scale @codepo8

About face… @codepo8 aka.ms/face-api

About face… @codepo8 ▪ Face rectangle / Landmarks ▪ Pose (pitch/roll/yaw) ▪ Smile ▪ Gender/Age ▪ Type of glasses ▪ Makeup (lips/eye) ▪ Emotion (anger, contempt, disgust, fear, happiness, neutral, sadness, surprise) ▪ Occlusion (forehead/eye/mouth) ▪ Facial hair (moustache/beard/sideburns) ▪ Attributes: Hair (invisible, bald, colour) aka.ms/face-api

Is this you? Are those also you? @codepo8 aka.ms/face-api

Is this your driver? @codepo8 youtube.com/watch?v=aEBi4OpXU4Q

Taking it too far? @codepo8 ntechlab.com

Detecting even more… @codepo8 https://apnews.com/bf75dd1c26c947b7826d270a16e2658a

Those trustworthy avatars… @codepo8 https://blog.insightdatascience.com/ generating-custom-photo-realistic-faces-using-ai-d170b1b59255

Those trustworthy avatars… @codepo8 https://blog.insightdatascience.com/ generating-custom-photo-realistic-faces-using-ai-d170b1b59255

Automated face mapping… @codepo8 https://github.com/SpiderLabs/social_mapper

Once you are known… @codepo8 https://github.com/SpiderLabs/social_mapper

Photo by Florian Ziegler flickr.com/photos/damndirty/40153024740/ AI for humans Andreas Dantz flickr.com/photos/szene/40193567250

I want people to appreciate AI, without giving up their data unwillingly… @codepo8

The best way to do this, is to stop selling it as magic, but as a tool… @codepo8

How AI can help humans… @codepo8 aka.ms/ai-for-good

Humans ▪ Messy and prone to mistakes ▪ Forget things and filter them by their biases Bots and computers… ▪ Make no mistakes, other than physical fatigue ▪ Never forget, don’t judge ▪ Bored when doing repetitive tasks ▪ Great at tedious, boring tasks ▪ When bored create more errors ▪ Repeat things with minor changes on iterations till a result is met ▪ Non-optimised communication, lots of nuances and misunderstanding @codepo8 ▪ Highly optimised, non-nuanced communication.

Humans ▪ Messy and prone to mistakes ▪ Forget things and filter them by their biases Bots and computers… ▪ Make no mistakes, other than physical fatigue ▪ Never forget, don’t judge ▪ Bored when doing repetitive tasks ▪ Great at tedious, boring tasks ▪ When bored create more errors ▪ Repeat things with minor changes on iterations till a result is met ▪ Non-optimised communication, lots of nuances and misunderstanding @codepo8 ▪ Highly optimised, non-nuanced communication.

We need data, so let’s make it joyful for humans to give us some @codepo8

Humans and Bots/Computers @codepo8 autodraw.com

Humans and Bots/Computers @codepo8 autodraw.com

Humans and Bots/Computers @codepo8 quickdraw.withgoogle.com

Humans and Bots/Computers @codepo8 google.com/recaptcha/intro

“Learning” from lots of images @codepo8 https://github.com/jantic/DeOldify

Humans and Bots/Computers aka.ms/nvidia-fix-image

Humans and Bots/Computers aka.ms/nvidia-fix-image

Humans and Bots/Computers aka.ms/nvidia-fix-image

Humans and Bots/Computers gandissect.csail.mit.edu/

Our toolkit for more human interfaces Natural language processing @codepo8 Computer Vision Sentiment analysis Speech conversion and analysis Moderation

Language and Writing @codepo8 ▪ Probably the oldest task on the web was translation ▪ This moved deeper into Natural Language Processing and Language Detection ▪ Using these, we can allow for human commands and finding out tasks by analyzing texts. “How far am I from the capital of Denmark?” “Where do I find a good restaurant around here?” “Show me documents I wrote five days ago with more than 600 words”

Computer Vision @codepo8 ▪ When text wasn’t cool enough, we added images to our web media ▪ Often we forget that not everyone can see them, and we leave them without alternative text ▪ This is where machine learning steps in to help turning an image into a dataset we can work with.

Vision and image analysis… instagram: @larryandanke @codepo8

Vision and image analysis… @codepo8

Vision and image analysis… @codepo8

Vision and image analysis… @codepo8 twitter.com/mixedhunty/status/980551155297157126

Vision and image analysis… @codepo8 #vision_api

Vision and image analysis… @codepo8 aka.ms/vision-api

Vision and image analysis… @codepo8 aka.ms/vision-api

Vision and image analysis… @codepo8 aka.ms/vision-api

Vision and image analysis… @codepo8 aka.ms/vision-api

Vision and image analysis… @codepo8 aka.ms/vision-api

Sentiment analysis @codepo8 ▪ Finding out the sentiment of a text, image or video can help with a lot of things ▪ You can navigate videos by only showing the happy parts ▪ You can detect which comment should be answered first by a help desk ▪ You can predict when drivers of cars get tired

▪ Audio interfaces are all the rage. ▪ You can allow hands-free control of devices ▪ You can have an “always on” system to help you out without having to interface with it ▪ It feels natural and has a massive Sci-Fi feeling – when it works. Speech @codepo8

Speech recognition @codepo8 aka.ms/text-to-speech

Turning sentences into commands @codepo8 luis.ai aka.ms/luis-api

Text to speech @codepo8 aka.ms/text-to-speech

Conversation as an interface @codepo8 aka.ms/conversation-ui

Speaker recognition @codepo8 aka.ms/speaker-recognition

Speaker recognition @codepo8 aka.ms/speaker-recognition

Moderation @codepo8 ▪ Some things are not meant to be consumed by people ▪ Computers don’t need counselling once they saw them – people should ▪ Known illegal and terrible content can be automatically removed

With great power comes great responsibility… @codepo8

Our responsibilities.. @codepo8 ▪ AI can be an amazing help for humans ▪ It does need transparency – if you use people as data sources, they need to know what and where it goes ▪ When people get information filtered by an algorithm, it should be an opt-in ▪ People need to have a chance to dispute when an algorithm tagged or disallowed them access.

Want to go deep? ▪ The Math behind ML ▪ The ethics of AI ▪ Working with Data using Python ▪ Machine Learning Models ▪ Deep Learning Models ▪ Reinforcement Learning Models ▪ Microsoft Professional Program Certificate in Artificial Intelligence aka.ms/learn-ai 10 courses, (8-16 hours each), 10 skills @codepo8

Want to go deep? skl.sh/christian Free with trial sign-up @codepo8

Who controls our data? Who benefits? @codepo8 ▪ With all this we need to make clear who has your data and where it goes. ▪ Wouldn’t it be great if we could do more on our devices? ▪ Much lower latency, better security, increased privacy ▪ Right now, this is only possible in native environments ▪ I want to change that – a W3C proposal to bring accelerated Machine Learning to the web in JavaScript

Who controls our data? Who benefits? w3.org/community/webmachinelearning @codepo8

Don’t forget to have fun! @codepo8

Suz Hinton @codepo8 github.com/noopkat/face-api-emoji-face

Categorising images by gesture @codepo8 http://pointerpointer.com

Find your moves @codepo8 https://experiments.withgoogle.com/move-mirror

Stay silly… @codepo8 Cassie Evans https://codepen.io/cassie-codes/pen/jKaVqo/

Help the human @codepo8 https://charliegerard.github.io/teachable-keyboard/

Artificial Intelligence @codepo8 https://charliegerard.github.io/teachable-keyboard/

Collaborate and share… @codepo8 Linda Liukas https://helloruby.com

Preparing the next generation @codepo8 Linda Liukas https://helloruby.com

Thanks! Chris Heilmann Christianheilmann.com Developer-evangelism.com @codepo8 http://inspirobot.me/

Welcome to Azure Saturday 2019 Munich #AzureSaturday 18.05.2019 – Microsoft Munich – azuresaturday.de — @azuresaturday