A presentation at Azure Saturday Munich 2019 in in Munich, Germany by Chris Heilmann
Welcome to Azure Saturday 2019 Munich #AzureSaturday 18.05.2019 – Microsoft Munich – azuresaturday.de — @azuresaturday
#AzureSaturday 18.05.2019 – Microsoft Munich – azuresaturday.de – @azuresaturday Building human interfaces powered by AI Speaker: Chris Heilmann
Building human interfaces powered by AI Chris Heilmann (@codepo8) November 2018
All resources: aka.ms/human-ai @codepo8
Let’s talk about “Artificial Intelligence” @codepo8
What is the difference between Machine Learning and Artificial Intelligence? @codepo8
Machine Learning is written in Python, JavaScript… Artificial Intelligence is written in PowerPoint. @codepo8
Artificial Intelligence @codepo8 ▪ Is nothing new – the concepts go back to the 50ies ▪ Is quite the hype and very often misattributed ▪ Is an umbrella term for a lot of math and science around repetition, pattern recognition and machine learning ▪ Got a huge boost because of availability of hardware ▪ Became much more feasible because of the availability of lots of data
Reminders of “genie in the bottle” @codepo8 ▪ Fulfills our wishes seemingly with invisible magic ▪ Useful, and feels too good to be true ▪ Once released, may have a dark, sinister edge to it ▪ Hard to put back into the bottle.
Let’s start with some predictions. @codepo8 ▪ AI is the number one growth market in IT – the others are cloud and security ▪ Machine Learning is already replacing thousands of jobs – boring, terrible jobs humans should not do ▪ This is also happening in IT – we are not invincible because we know hot to exit Vim
Let’s start with some predictions. @codepo8 ▪ There is no stopping this – it is just too convenient ▪ The amount of data we create (actively or by triggering sensors) demands machines to whittle it down for us to make it consumable by humans ▪ If we as developers and decision makers in IT don’t take ownership and lead with good, ethical examples, we’ll throw away decades of work democratising computing
The machines are watching… Florian Ziegler flickr.com/photos/damndirty/41263240134
Social Credit System @codepo8 https://futurism.com/china-social-credit-system-rate-human-value/
Big brother is redundant… @codepo8 ▪ Everything we do online is monitored and recorded ▪ We often don’t realise that our data is how we pay for “free” services ▪ We’re happy to use systems that record all the time in exchange for convenience ▪ Often people don’t realise just how dangerous this can be in the wrong hands.
Everything counts in large amounts @codepo8 ▪ We create a massive amount of information – actively and without our knowledge. ▪ It is tough to make that amount of information consumable again. ▪ That’s why we have computers ▪ With cloud computing, on demand processing and advances in hardware we’re faster than ever.
Leaving invisible marks… @codepo8 ▪ By using other people’s machines and infrastructure, we leave traces ▪ This allows companies to recognise us, and accumulates a usage history ▪ This leads to better results, but can leak data ▪ We should have more transparency about what digital legacy we left behind.
Are machines friend or foe? Florian Ziegler flickr.com/photos/damndirty/40153024740/
Artificial Intelligence Myths @codepo8 ▪ AI can’t replace a thinking, creative human ▪ AI can not magically fill gaps with perfect information – it can only compare and assume ▪ AI doesn’t learn in a creative fashion. It makes no assumptions ▪ AI has no morals and ethics, but – used wrongly – it can amplify our biases
Machines can be great tools or weapons… @codepo8 ▪ Machine Learning is all about returning assumptions ▪ We don’t get any definitive truth from algorithms, we get answers to our questions ▪ AI can answer questions, but it is up to you to ask good questions – generic questions yield assumed results.
Unguided or supervised AI… @codepo8 http://inspirobot.me
It can be demanding @codepo8 http://inspirobot.me
It can mix up needs… @codepo8 http://inspirobot.me
It can be overly excited… @codepo8 http://inspirobot.me
It can be a good warning… @codepo8 http://inspirobot.me
It can be painfully humbling… @codepo8 http://inspirobot.me
Prophetic, even? @codepo8 http://inspirobot.me
Passive aggressive towards humans… @codepo8
It can be adoringly cute… @codepo8 https://twitter.com/eron_gj/status/967672260147470336
Whilst being actually kick-ass @codepo8 https://www.youtube.com/watch?v=gn4nRCC9TwQ
Machines can be great tools or weapons… @codepo8 ▪ Untrained and limited data leads to terrible and biased AI results ▪ It is very easy to get either wrong deductions or false positives ▪ AI is as intelligent and good as the people who apply it
Machine learning helps us in a few ways… @codepo8 ▪ Recommendation ▪ Prediction ▪ Classification ▪ Clustering ▪ Generation
Machines ploughing through lots of data for you. Recommendation @codepo8 ▪ “I feel lucky” moments ▪ Slack finding people in your organization ▪ Intelligent inboxes ▪ Automated photo optimization ▪ Automated tagging and alternative text: “Image may contain”
You’re doing this – you probably want this as the next thing Prediction @codepo8 ▪ Text autocompletion ▪ Task offerings ▪ Image tooling – adding photos to a collage ▪ Creating albums ▪ Offering similar music and videos ▪ Offering products that match
Sort things by what humans told you what they are and scale it up Classification @codepo8 ▪ Google surveys offering the right form elements for a question ▪ Detecting faces and asking for more information ▪ Finding anomalies in health scans and doing the same for all the ones in the system
Find own patterns and collate them Clustering @codepo8 ▪ Photo tagging and ordering ▪ Document analysis ▪ Comment filtering and triaging ▪ Video optimisation dependent on content.
Allow the machine to create things Generation @codepo8 ▪ Art style matching ▪ Generated articles from fact collection ▪ Synthesised music ▪ Filling content with tagged information (grass, houses, brick, etc…) ▪ React to human input
We need to find our place on the scale @codepo8
About face… @codepo8 aka.ms/face-api
About face… @codepo8 ▪ Face rectangle / Landmarks ▪ Pose (pitch/roll/yaw) ▪ Smile ▪ Gender/Age ▪ Type of glasses ▪ Makeup (lips/eye) ▪ Emotion (anger, contempt, disgust, fear, happiness, neutral, sadness, surprise) ▪ Occlusion (forehead/eye/mouth) ▪ Facial hair (moustache/beard/sideburns) ▪ Attributes: Hair (invisible, bald, colour) aka.ms/face-api
Is this you? Are those also you? @codepo8 aka.ms/face-api
Is this your driver? @codepo8 youtube.com/watch?v=aEBi4OpXU4Q
Taking it too far? @codepo8 ntechlab.com
Detecting even more… @codepo8 https://apnews.com/bf75dd1c26c947b7826d270a16e2658a
Those trustworthy avatars… @codepo8 https://blog.insightdatascience.com/ generating-custom-photo-realistic-faces-using-ai-d170b1b59255
Those trustworthy avatars… @codepo8 https://blog.insightdatascience.com/ generating-custom-photo-realistic-faces-using-ai-d170b1b59255
Automated face mapping… @codepo8 https://github.com/SpiderLabs/social_mapper
Once you are known… @codepo8 https://github.com/SpiderLabs/social_mapper
Photo by Florian Ziegler flickr.com/photos/damndirty/40153024740/ AI for humans Andreas Dantz flickr.com/photos/szene/40193567250
I want people to appreciate AI, without giving up their data unwillingly… @codepo8
The best way to do this, is to stop selling it as magic, but as a tool… @codepo8
How AI can help humans… @codepo8 aka.ms/ai-for-good
Humans ▪ Messy and prone to mistakes ▪ Forget things and filter them by their biases Bots and computers… ▪ Make no mistakes, other than physical fatigue ▪ Never forget, don’t judge ▪ Bored when doing repetitive tasks ▪ Great at tedious, boring tasks ▪ When bored create more errors ▪ Repeat things with minor changes on iterations till a result is met ▪ Non-optimised communication, lots of nuances and misunderstanding @codepo8 ▪ Highly optimised, non-nuanced communication.
Humans ▪ Messy and prone to mistakes ▪ Forget things and filter them by their biases Bots and computers… ▪ Make no mistakes, other than physical fatigue ▪ Never forget, don’t judge ▪ Bored when doing repetitive tasks ▪ Great at tedious, boring tasks ▪ When bored create more errors ▪ Repeat things with minor changes on iterations till a result is met ▪ Non-optimised communication, lots of nuances and misunderstanding @codepo8 ▪ Highly optimised, non-nuanced communication.
We need data, so let’s make it joyful for humans to give us some @codepo8
Humans and Bots/Computers @codepo8 autodraw.com
Humans and Bots/Computers @codepo8 autodraw.com
Humans and Bots/Computers @codepo8 quickdraw.withgoogle.com
Humans and Bots/Computers @codepo8 google.com/recaptcha/intro
“Learning” from lots of images @codepo8 https://github.com/jantic/DeOldify
Humans and Bots/Computers aka.ms/nvidia-fix-image
Humans and Bots/Computers aka.ms/nvidia-fix-image
Humans and Bots/Computers aka.ms/nvidia-fix-image
Humans and Bots/Computers gandissect.csail.mit.edu/
Our toolkit for more human interfaces Natural language processing @codepo8 Computer Vision Sentiment analysis Speech conversion and analysis Moderation
Language and Writing @codepo8 ▪ Probably the oldest task on the web was translation ▪ This moved deeper into Natural Language Processing and Language Detection ▪ Using these, we can allow for human commands and finding out tasks by analyzing texts. “How far am I from the capital of Denmark?” “Where do I find a good restaurant around here?” “Show me documents I wrote five days ago with more than 600 words”
Computer Vision @codepo8 ▪ When text wasn’t cool enough, we added images to our web media ▪ Often we forget that not everyone can see them, and we leave them without alternative text ▪ This is where machine learning steps in to help turning an image into a dataset we can work with.
Vision and image analysis… instagram: @larryandanke @codepo8
Vision and image analysis… @codepo8
Vision and image analysis… @codepo8
Vision and image analysis… @codepo8 twitter.com/mixedhunty/status/980551155297157126
Vision and image analysis… @codepo8 #vision_api
Vision and image analysis… @codepo8 aka.ms/vision-api
Vision and image analysis… @codepo8 aka.ms/vision-api
Vision and image analysis… @codepo8 aka.ms/vision-api
Vision and image analysis… @codepo8 aka.ms/vision-api
Vision and image analysis… @codepo8 aka.ms/vision-api
Sentiment analysis @codepo8 ▪ Finding out the sentiment of a text, image or video can help with a lot of things ▪ You can navigate videos by only showing the happy parts ▪ You can detect which comment should be answered first by a help desk ▪ You can predict when drivers of cars get tired
▪ Audio interfaces are all the rage. ▪ You can allow hands-free control of devices ▪ You can have an “always on” system to help you out without having to interface with it ▪ It feels natural and has a massive Sci-Fi feeling – when it works. Speech @codepo8
Speech recognition @codepo8 aka.ms/text-to-speech
Turning sentences into commands @codepo8 luis.ai aka.ms/luis-api
Text to speech @codepo8 aka.ms/text-to-speech
Conversation as an interface @codepo8 aka.ms/conversation-ui
Speaker recognition @codepo8 aka.ms/speaker-recognition
Speaker recognition @codepo8 aka.ms/speaker-recognition
Moderation @codepo8 ▪ Some things are not meant to be consumed by people ▪ Computers don’t need counselling once they saw them – people should ▪ Known illegal and terrible content can be automatically removed
With great power comes great responsibility… @codepo8
Our responsibilities.. @codepo8 ▪ AI can be an amazing help for humans ▪ It does need transparency – if you use people as data sources, they need to know what and where it goes ▪ When people get information filtered by an algorithm, it should be an opt-in ▪ People need to have a chance to dispute when an algorithm tagged or disallowed them access.
Want to go deep? ▪ The Math behind ML ▪ The ethics of AI ▪ Working with Data using Python ▪ Machine Learning Models ▪ Deep Learning Models ▪ Reinforcement Learning Models ▪ Microsoft Professional Program Certificate in Artificial Intelligence aka.ms/learn-ai 10 courses, (8-16 hours each), 10 skills @codepo8
Want to go deep? skl.sh/christian Free with trial sign-up @codepo8
Who controls our data? Who benefits? @codepo8 ▪ With all this we need to make clear who has your data and where it goes. ▪ Wouldn’t it be great if we could do more on our devices? ▪ Much lower latency, better security, increased privacy ▪ Right now, this is only possible in native environments ▪ I want to change that – a W3C proposal to bring accelerated Machine Learning to the web in JavaScript
Who controls our data? Who benefits? w3.org/community/webmachinelearning @codepo8
Don’t forget to have fun! @codepo8
Suz Hinton @codepo8 github.com/noopkat/face-api-emoji-face
Categorising images by gesture @codepo8 http://pointerpointer.com
Find your moves @codepo8 https://experiments.withgoogle.com/move-mirror
Stay silly… @codepo8 Cassie Evans https://codepen.io/cassie-codes/pen/jKaVqo/
Help the human @codepo8 https://charliegerard.github.io/teachable-keyboard/
Artificial Intelligence @codepo8 https://charliegerard.github.io/teachable-keyboard/
Collaborate and share… @codepo8 Linda Liukas https://helloruby.com
Preparing the next generation @codepo8 Linda Liukas https://helloruby.com
Thanks! Chris Heilmann Christianheilmann.com Developer-evangelism.com @codepo8 http://inspirobot.me/
There is no question that we are living in the age of automation and machine learning. Humans and sensors create far too much data for humans to comprehend which is why we need machines to make them digestible for us. The problem is that machines don’t have any ethics and technology is still seen as magic by people who give away far too much of their personal data without knowing who listens. To make the AI revolution work we need to build ethical systems and install an ownership in users.
The following resources were mentioned during the presentation or are useful additional information.
for free. You
can too.