A presentation at GDG Johannesburg in in South Africa by Rebecca Franks
Android Developers Guide to Machine Learning ! With MLKit, Tensorflow & Firebase "
Rebecca Franks @riggaroo
Google Developer Expert
Android @ Over
Pluralsight Author
GDG Johannesburg Organiser
Who considers themselves an expert in Machine Learning?
What is Machine Learning?
Machine learning is an application of Artificial Intelligence in which we input a lot of data and let the machines learn
“by themselves”
Face detection $ % Works offline
Demo
& ⛩
Landmark detection ( )
Demo
Image Labelling * Works offline
Barcode scanning + Works offline
FirebaseVisionBarcodeDetectorOptions . Builder ()
. setBarcodeFormats ( FirebaseVisionBarcode . FORMAT_QR_CODE )
. build ()
val image = FirebaseVisionImage.fromBitmap(bitmap) val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector.detectInImage(image) .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it)) var result = String()
FirebaseVisionImage . fromBitmap ( bitmap )
FirebaseVision . getInstance () . getVisionBarcodeDetector ( options )
detector.detectInImage(image) .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it)) var result = String() it.forEach { result += "VALUE TYPE: ${it.valueType} Raw Value: ${it.rawValue}" textResult.postValue(result) }
val image = FirebaseVisionImage.fromBitmap(bitmap) val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector . detectInImage ( image )
. addOnSuccessListener {
processedBitmap
. postValue ( barcodeProcessor . drawBoxes ( bitmap , it ))
String ()
it
. forEach {
result
+=
"VALUE TYPE: ${it.valueType} Raw Value: ${it.rawValue}"
textResult
. postValue ( result )
}
}. addOnFailureListener {
textResult
. postValue ( it . message )
}
OCR ,
Works offline
On Device vs Cloud
private fun doOcrDetection ( bitmap :
Bitmap ){
val detector
=
FirebaseVision . getInstance ()
FirebaseVisionImage . fromBitmap ( bitmap )
detector.detectInImage(firebaseImage) .addOnSuccessListener { processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result) }
private fun doOcrDetection(bitmap: Bitmap){ val detector = FirebaseVision.getInstance() .visionTextDetector val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap) detector . detectInImage ( firebaseImage )
. addOnSuccessListener {
processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result) }
}
. addOnFailureListener {
Toast.makeText(/../“Error detecting Text $it”/../)
}
}
private fun doOcrDetection(bitmap: Bitmap){ val detector = FirebaseVision.getInstance() .visionTextDetector val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap) detector . detectInImage ( firebaseImage )
. addOnSuccessListener {
processedBitmap
. postValue ( ocrProcessor . drawBoxes ( bitmap , it ))
String ()
it
. blocks . forEach {
result
+=
" "
it . text textResult . postValue ( result )
}
}
. addOnFailureListener {
Toast . makeText ( /../ “Error detecting Text $it” /../ )
}
}
Custom Tensorflow Models . / Works offline
TensorFlow
Retrain existing model 0
mobilenet_v 1
What if I could tell what kind of chips I was eating?
Gather
Training
Data
FFMPEG
Folders of
Images
Retrain with
new images
Optimize
for mobile
Embed in app
Store in Firebase
App
uses
model
Gather training
data
Export to Images using ffmpeg ffmpeg -i flings.mp4 flings/flings_%04d.jpg
Folders of images
Retrain with new images python -m scripts.retrain
--bottleneck_dir=tf_files/bottlenecks
--how_many_training_steps=500
--model_dir=tf_files/models/
--summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}"
--output_graph=tf_files/
retrained_graph.pb
--output_labels=tf_files/retrained_labels.txt
--architecture="${ARCHITECTURE}"
--image_dir=training_data/
south_african_chips
Optimize for mobile
bazel-bin/tensorflow/contrib/lite/toco/toco
--input_file=AgencyDay/
retrained_graph.pb
--output_file=AgencyDay/ chips_optimized_graph.tflite
--input_format=TENSORFLOW_GRAPHDEF
--output_format=TFLITE
--input_shape=1,${IMAGE_SIZE},${IMAGE_SIZE},3
--input_array=input
--output_array=final_result
--inference_type=FLOAT
--input_data_type=FLOAT
Insert into App https://codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/
https://github.com/googlecodelabs/tensorflow-for-poets-2
Nik Nak 1
Or Not 2
bit.ly/mlkit-riggaroo
What if our model changes?
Ship an app update 3
And hope that people download it 4
Host on Firebase "
Updates automatically downloaded
FirebaseCloudModelSource . Builder ( "my_cloud_model" )
. enableModelUpdates ( true )
. setInitialDownloadConditions ( conditions )
. setUpdatesDownloadConditions ( conditions )
. build () FirebaseModelManager . getInstance () . registerCloudModelSource ( cloudSource ) ……
g.co/codelabs/mlkit-android-custom-model
You don’t need to be a ML Expert to take advantage of ML in your apps! !
Thank you!
https://codelabs.developers.google.com/ codelabs/tensorflow-for-poets
https://codelabs.developers.google.com/ codelabs/tensorflow-for-poets-2-tflite/
https://codelabs.developers.google.com/ codelabs/mlkit-android-custom-model/ #0
https://github.com/riggaroo/android- demo-mlkit
In this presentation, Rebecca demonstrates how to use the new Firebase ML Kit and how to train your own TensorFlow Lite custom model.
The following resources were mentioned during the presentation or are useful additional information.
Recording of the event
Here’s what was said about this presentation on Twitter.
@GDGJohannesburg it's always fascinating how @riggaroo can take a complicated topic and make it seem simple. #MLkit #gdgjhb
— Velaphi Mathebula (@VejaTheGreat) August 8, 2018
Time to detect faces 😊😊... Machine Leraning everyday! Its no more a practice for specialists. @GDGJohannesburg #gdgjhb pic.twitter.com/861QX4PPxV
— Potsane Mohale (@MohalePotsane) August 8, 2018
Next up, @riggaroo with an "Android Developer's Guide to Machine Learning" #GDGJhb pic.twitter.com/CWyHd561Ex
— GDG Johannesburg (@GDGJohannesburg) August 8, 2018
Fun talk about ML Kit and TensorFlow Lite by @riggaroo at the Joburg Google Agency Day. Great demos! pic.twitter.com/ZIdl6Pd0mn
— Rishal Hurbans (@RishalHurbans) June 28, 2018
Did my demo showcasing my Nik Nak or Not app that I built my own custom retrained @TensorFlow model 🍟
— Rebecca Franks (@riggaroo) June 28, 2018
You can download the demo apps from here:https://t.co/1GO3mvtTwl
Slides from my talk: https://t.co/I9EOyiCYJA@GoogleDevExpert #GDE #AndroidDev #mlkit pic.twitter.com/pYJ5S95MAp
Thanks Rebecca Franks @riggaroo and Johann du Toit @signedness for a very informative and "mind opening" talk on Machine Learning (with a focus on Google products)! And of course @GDGCapeTown's @NickRout for providing the awesome @over meetup space and refreshments! https://t.co/2udxmx3CuQ
— Andre Rademan (@andrerademan) August 2, 2018