A presentation at GDG Johannesburg in in South Africa by Rebecca Franks
Android Developers Guide to Machine Learning ! With MLKit, Tensorflow & Firebase "
Rebecca Franks @riggaroo
Google Developer Expert
Android @ Over
Pluralsight Author
GDG Johannesburg Organiser
Who considers themselves an expert in Machine Learning?
What is Machine Learning?
Machine learning is an application of Artificial Intelligence in which we input a lot of data and let the machines learn
“by themselves”
Face detection $ % Works offline
Demo
& ⛩
Landmark detection ( )
Demo
Image Labelling * Works offline
Barcode scanning + Works offline
FirebaseVisionBarcodeDetectorOptions . Builder ()
. setBarcodeFormats ( FirebaseVisionBarcode . FORMAT_QR_CODE )
. build ()
val image = FirebaseVisionImage.fromBitmap(bitmap) val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector.detectInImage(image) .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it)) var result = String()
FirebaseVisionImage . fromBitmap ( bitmap )
FirebaseVision . getInstance () . getVisionBarcodeDetector ( options )
detector.detectInImage(image) .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it)) var result = String() it.forEach { result += "VALUE TYPE: ${it.valueType} Raw Value: ${it.rawValue}" textResult.postValue(result) }
val image = FirebaseVisionImage.fromBitmap(bitmap) val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector . detectInImage ( image )
. addOnSuccessListener {
processedBitmap
. postValue ( barcodeProcessor . drawBoxes ( bitmap , it ))
String ()
it
. forEach {
result
+=
"VALUE TYPE: ${it.valueType} Raw Value: ${it.rawValue}"
textResult
. postValue ( result )
}
}. addOnFailureListener {
textResult
. postValue ( it . message )
}
OCR ,
Works offline
On Device vs Cloud
private fun doOcrDetection ( bitmap :
Bitmap ){
val detector
=
FirebaseVision . getInstance ()
FirebaseVisionImage . fromBitmap ( bitmap )
detector.detectInImage(firebaseImage) .addOnSuccessListener { processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result) }
private fun doOcrDetection(bitmap: Bitmap){ val detector = FirebaseVision.getInstance() .visionTextDetector val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap) detector . detectInImage ( firebaseImage )
. addOnSuccessListener {
processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result) }
}
. addOnFailureListener {
Toast.makeText(/../“Error detecting Text $it”/../)
}
}
private fun doOcrDetection(bitmap: Bitmap){ val detector = FirebaseVision.getInstance() .visionTextDetector val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap) detector . detectInImage ( firebaseImage )
. addOnSuccessListener {
processedBitmap
. postValue ( ocrProcessor . drawBoxes ( bitmap , it ))
String ()
it
. blocks . forEach {
result
+=
" "
it . text textResult . postValue ( result )
}
}
. addOnFailureListener {
Toast . makeText ( /../ “Error detecting Text $it” /../ )
}
}
Custom Tensorflow Models . / Works offline
TensorFlow
Retrain existing model 0
mobilenet_v 1
What if I could tell what kind of chips I was eating?
Gather
Training
Data
FFMPEG
Folders of
Images
Retrain with
new images
Optimize
for mobile
Embed in app
Store in Firebase
App
uses
model
Gather training
data
Export to Images using ffmpeg ffmpeg -i flings.mp4 flings/flings_%04d.jpg
Folders of images
Retrain with new images python -m scripts.retrain
--bottleneck_dir=tf_files/bottlenecks
--how_many_training_steps=500
--model_dir=tf_files/models/
--summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}"
--output_graph=tf_files/
retrained_graph.pb
--output_labels=tf_files/retrained_labels.txt
--architecture="${ARCHITECTURE}"
--image_dir=training_data/
south_african_chips
Optimize for mobile
bazel-bin/tensorflow/contrib/lite/toco/toco
--input_file=AgencyDay/
retrained_graph.pb
--output_file=AgencyDay/ chips_optimized_graph.tflite
--input_format=TENSORFLOW_GRAPHDEF
--output_format=TFLITE
--input_shape=1,${IMAGE_SIZE},${IMAGE_SIZE},3
--input_array=input
--output_array=final_result
--inference_type=FLOAT
--input_data_type=FLOAT
Insert into App https://codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/
https://github.com/googlecodelabs/tensorflow-for-poets-2
Nik Nak 1
Or Not 2
bit.ly/mlkit-riggaroo
What if our model changes?
Ship an app update 3
And hope that people download it 4
Host on Firebase "
Updates automatically downloaded
FirebaseCloudModelSource . Builder ( "my_cloud_model" )
. enableModelUpdates ( true )
. setInitialDownloadConditions ( conditions )
. setUpdatesDownloadConditions ( conditions )
. build () FirebaseModelManager . getInstance () . registerCloudModelSource ( cloudSource ) ……
g.co/codelabs/mlkit-android-custom-model
You don’t need to be a ML Expert to take advantage of ML in your apps! !
Thank you!
https://codelabs.developers.google.com/ codelabs/tensorflow-for-poets
https://codelabs.developers.google.com/ codelabs/tensorflow-for-poets-2-tflite/
https://codelabs.developers.google.com/ codelabs/mlkit-android-custom-model/ #0
https://github.com/riggaroo/android- demo-mlkit
In this presentation, Rebecca demonstrates how to use the new Firebase ML Kit and how to train your own TensorFlow Lite custom model.
The following resources were mentioned during the presentation or are useful additional information.
Recording of the event
Here’s what was said about this presentation on Twitter.
for free. You
can too.