A presentation at Codemotion Berlin in in Berlin, Germany by Chris Heilmann
Berlin | November 20 - 21, 2018 Building Human Interfaces powered by AI Christian Heilmann
Building human interfaces powered by AI Chris Heilmann (@codepo8) November 2018
All resources: aka.ms/human-ai @codepo8
Let’s talk about “Artificial Intelligence” @codepo8
Artificial Intelligence @codepo8 ▪ Is nothing new – the concepts go back to the 50ies ▪ Is quite the hype and very often misattributed ▪ Is an umbrella term for a lot of math and science around repetition, pattern recognition and machine learning ▪ Got a huge boost because of availability of hardware
The machines are watching… Florian Ziegler flickr.com/photos/damndirty/41263240134
Big brother is redundant… @codepo8 ▪ Everything we do online is monitored and recorded ▪ We often don’t realise that our data is how we pay for “free” services ▪ We’re happy to use systems that record all the time in exchange for convenience ▪ Often people don’t realise just how dangerous this can be in the wrong hands.
Everything counts in large amounts @codepo8 ▪ We create a massive amount of information – actively and without our knowledge. ▪ It is tough to make that amount of information consumable again. ▪ That’s why we have computers ▪ With cloud computing, on demand processing and advances in hardware we’re faster than ever.
Leaving invisible marks… @codepo8 ▪ By using other people’s machines and infrastructure, we leave traces ▪ This allows companies to recognise us, and accumulates a usage history ▪ This leads to better results, but can leaks data ▪ We should have more transparency about what digital legacy we left behind.
Are machines friend or foe? Florian Ziegler flickr.com/photos/damndirty/40153024740/
Artificial Intelligence Myths @codepo8 ▪ AI can’t replace a thinking, creative human ▪ AI can not magically fill gaps with perfect information – it can only compare and assume ▪ AI doesn’t learn in a creative fashion. It makes no assumptions ▪ AI has no morals and ethics, but – used wrongly – it can amplify our biases
Machines can be great tools or weapons… @codepo8 ▪ Machine Learning is all about returning assumptions ▪ We don’t get any definitive truth from algorithms, we get answers to our questions ▪ AI can answer questions, but it is up to you to ask good questions – generic questions yield assumed results.
Machines can be great tools or weapons… @codepo8 ▪ Untrained and limited data leads to terrible and biased AI results ▪ It is very easy to get either wrong deductions or false positives ▪ AI is as intelligent and good as the people who apply it
About face… @codepo8 aka.ms/face-api
About face… @codepo8 ▪ Face rectangle / Landmarks ▪ Pose (pitch/roll/yaw) ▪ Smile ▪ Gender/Age ▪ Type of glasses ▪ Makeup (lips/eye) ▪ Emotion (anger, contempt, disgust, fear, happiness, neutral, sadness, surprise) ▪ Occlusion (forehead/eye/mouth) ▪ Facial hair (moustache/beard/sideburns) ▪ Attributes: Hair (invisible, bald, colour) aka.ms/face-api
Is this you? Are those also you? @codepo8 aka.ms/face-api
Is this your driver? @codepo8 youtube.com/watch?v=aEBi4OpXU4Q
Taking it too far? @codepo8 ntechlab.com
Detecting even more… @codepo8 https://apnews.com/bf75dd1c26c947b7826d270a16e2658a
Those trustworthy avatars… @codepo8 https://blog.insightdatascience.com/ generating-custom-photo-realistic-faces-using-ai-d170b1b59255
Those trustworthy avatars… @codepo8 https://blog.insightdatascience.com/ generating-custom-photo-realistic-faces-using-ai-d170b1b59255
Automated face mapping… @codepo8 https://github.com/SpiderLabs/social_mapper
Once you are known… @codepo8 https://github.com/SpiderLabs/social_mapper
Photo by Florian Ziegler flickr.com/photos/damndirty/40153024740/ AI for humans Andreas Dantz flickr.com/photos/szene/40193567250
How AI can help humans… @codepo8 aka.ms/ai-for-good
How AI can help humans… @codepo8 ▪ Automation ▪ Error prevention ▪ Data reduction / Muffling the noise ▪ Prediction based on historical data ▪ Ploughing through massive amounts of data ▪ Creating more human interfaces
How AI can help humans… @codepo8 ▪ Automation ▪ Error prevention ▪ Data reduction / Muffling the noise ▪ Prediction based on historical data ▪ Ploughing through massive amounts of data ▪ Creating more human interfaces
Humans ▪ Messy and prone to mistakes ▪ Forget things and filter them by their biases Bots and computers… ▪ Make no mistakes, other than physical fatigue ▪ Never forget, don’t judge ▪ Bored when doing repetitive tasks ▪ Great at tedious, boring tasks ▪ When bored create more errors ▪ Repeat things with minor changes on iterations till a result is met ▪ Non-optimised communication, lots of nuances and misunderstanding @codepo8 ▪ Highly optimised, non-nuanced communication.
Humans ▪ Messy and prone to mistakes ▪ Forget things and filter them by their biases Bots and computers… ▪ Make no mistakes, other than physical fatigue ▪ Never forget, don’t judge ▪ Bored when doing repetitive tasks ▪ Great at tedious, boring tasks ▪ When bored create more errors ▪ Repeat things with minor changes on iterations till a result is met ▪ Non-optimised communication, lots of nuances and misunderstanding @codepo8 ▪ Highly optimised, non-nuanced communication.
Humans Bots and computers… ▪ Messy and prone to mistakes ▪ Forget things and filter them by their biases ▪ Bored when doing repetitive tasks ▪ When bored create more errors ▪ Non-optimised communication, lots of nuances and misunderstanding @codepo8 Data Insights Patterns ▪ Make no mistakes, other than physical fatigue ▪ Never forget, don’t judge ▪ Great at tedious, boring tasks ▪ Repeat things with minor changes on iterations till a result is met ▪ Highly optimised, non-nuanced communication.
Humans and Bots/Computers @codepo8 autodraw.com
Humans and Bots/Computers @codepo8 autodraw.com
Humans and Bots/Computers @codepo8 quickdraw.withgoogle.com
Humans and Bots/Computers @codepo8 google.com/recaptcha/intro
“Learning” from lots of images @codepo8 https://github.com/jantic/DeOldify
Humans and Bots/Computers aka.ms/nvidia-fix-image
Humans and Bots/Computers aka.ms/nvidia-fix-image
Humans and Bots/Computers aka.ms/nvidia-fix-image
Intelligent, responsive systems ▪ AI services offer us lots of data to compare our users’ input with ▪ Thus our users don’t need to speak computer but be human instead ▪ We can prevent them from making mistakes ▪ We can help getting around physical barriers Google: cloud.google.com/products/machine-learning Amazon: aws.amazon.com/machine-learning @codepo8 Microsoft: azure.microsoft.com/en-us/services/cognitive-services
Our toolkit for more human interfaces Natural language processing @codepo8 Computer Vision Sentiment analysis Speech conversion and analysis Moderation
Language and Writing @codepo8 ▪ Probably the oldest task on the web was translation ▪ This moved deeper into Natural Language Processing and Language Detection ▪ Using these, we can allow for human commands and finding out tasks by analyzing texts. “How far am I from the capital of Denmark?” “Where do I find a good restaurant around here?” “Show me documents I wrote five days ago with more than 600 words”
Computer Vision @codepo8 ▪ When text wasn’t cool enough, we added images to our web media ▪ Often we forget that not everyone can see them, and we leave them without alternative text ▪ This is where machine learning steps in to help turning an image into a dataset we can work with.
Vision and image analysis… instagram: @larryandanke @codepo8
Vision and image analysis… @codepo8
Vision and image analysis… @codepo8
Vision and image analysis… @codepo8 twitter.com/mixedhunty/status/980551155297157126
Vision and image analysis… @codepo8 #vision_api
Vision and image analysis… @codepo8 aka.ms/vision-api
Vision and image analysis… @codepo8 aka.ms/vision-api
Vision and image analysis… @codepo8 aka.ms/vision-api
Vision and image analysis… @codepo8 aka.ms/vision-api
Vision and image analysis… @codepo8 aka.ms/vision-api
Sentiment analysis @codepo8 ▪ Finding out the sentiment of a text, image or video can help with a lot of things ▪ You can navigate videos by only showing the happy parts ▪ You can detect which comment should be answered first by a help desk ▪ You can predict when drivers of cars get tired
▪ Audio interfaces are all the rage. ▪ You can allow hands-free control of devices ▪ You can have an “always on” system to help you out without having to interface with it ▪ It feels natural and has a massive Sci-Fi feeling – when it works. Speech @codepo8
Speech recognition @codepo8 aka.ms/text-to-speech
Turning sentences into commands @codepo8 luis.ai aka.ms/luis-api
Text to speech @codepo8 aka.ms/text-to-speech
Conversation as an interface @codepo8 aka.ms/conversation-ui
Speaker recognition @codepo8 aka.ms/speaker-recognition
Speaker recognition @codepo8 aka.ms/speaker-recognition
Moderation @codepo8 ▪ Some things are not meant to be consumed by people ▪ Computers don’t need counselling once they saw them – people should ▪ Known illegal and terrible content can be automatically removed
With great power comes great responsibility… @codepo8
Our responsibilities.. @codepo8 ▪ AI can be an amazing help for humans ▪ It does need transparency – if you use people as data sources, they need to know what and where it goes ▪ When people get information filtered by an algorithm, it should be an opt-in ▪ People need to have a chance to dispute when an algorithm tagged or disallowed them access.
Want to go deep? ▪ The Math behind ML ▪ The ethics of AI ▪ Working with Data using Python ▪ Machine Learning Models ▪ Deep Learning Models ▪ Reinforcement Learning Models ▪ Microsoft Professional Program Certificate in Artificial Intelligence aka.ms/learn-ai 10 courses, (8-16 hours each), 10 skills @codepo8
Want to go deep? skl.sh/christian Free with trial sign-up @codepo8
Don’t forget to have fun! @codepo8
Suz Hinton @codepo8 github.com/noopkat/face-api-emoji-face
Categorising images by gesture @codepo8 http://pointerpointer.com
Find your moves @codepo8 https://experiments.withgoogle.com/move-mirror
Stay silly… @codepo8 Cassie Evans https://codepen.io/cassie-codes/pen/jKaVqo/
Help the human @codepo8 https://charliegerard.github.io/teachable-keyboard/
Artificial Intelligence @codepo8 https://charliegerard.github.io/teachable-keyboard/
Preparing the next generation @codepo8 Linda Liukas https://helloruby.com
Thanks! Chris Heilmann Christianheilmann.com Developer-evangelism.com @codepo8
We are smack in the middle of a hype about AI and whilst we wonder if a robot will take our job tomorrow or a badly configured self-driving car will run us over we're forgetting that AI can be a great way for humans to deal with machines and interfaces. Let's use Natural Language Processing to allow for clever search boxes, Artificial Vision to create automatic alternative text and facial recognition for automatic tagging and collation. The clever bits of Facebook are now available to all of us as APIs. Let's help humans embrace the revolution that is deep learning by showing them the benefits
The following resources were mentioned during the presentation or are useful additional information.
Here’s what was said about this presentation on social media.
for free. You
can too.