Android Developers Guide to Machine Learning ! With MLKit, Tensorflow & Firebase "

Rebecca Franks @riggaroo

Google Developer Expert
Android @ Over Pluralsight Author GDG Johannesburg Organiser

Who considers themselves an expert in Machine Learning?

What is Machine Learning?

Machine learning   is an application of Artificial Intelligence in which we input a lot of data and let the machines learn

“by themselves”

Face detection $ % Works offline

Demo

& ⛩

Landmark detection ( )

Demo

Image Labelling * Works offline

Barcode scanning + Works offline

val options

FirebaseVisionBarcodeDetectorOptions . Builder ()

. setBarcodeFormats ( FirebaseVisionBarcode . FORMAT_QR_CODE )

. build ()

val image = FirebaseVisionImage.fromBitmap(bitmap) val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector.detectInImage(image) .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it)) var result = String()

val options = FirebaseVisionBarcodeDetectorOptions.Builder() .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_QR_CODE) .build() val image

FirebaseVisionImage . fromBitmap ( bitmap )

val detector

FirebaseVision . getInstance () . getVisionBarcodeDetector ( options )

detector.detectInImage(image) .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it)) var result = String() it.forEach { result += "VALUE TYPE: ${it.valueType} Raw Value: ${it.rawValue}" textResult.postValue(result) }

val image = FirebaseVisionImage.fromBitmap(bitmap) val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector . detectInImage ( image )

. addOnSuccessListener {

    processedBitmap

. postValue ( barcodeProcessor . drawBoxes ( bitmap , it ))

var result

String ()

    it

. forEach {

        result

+=

"VALUE TYPE: ${it.valueType} Raw Value: ${it.rawValue}"

        textResult

. postValue ( result )

}

}. addOnFailureListener {

    textResult

. postValue ( it . message )

}

OCR ,

Works offline

On Device vs Cloud

private fun doOcrDetection ( bitmap :

Bitmap ){

val detector

=

FirebaseVision . getInstance ()

. visionTextDetector val firebaseImage

FirebaseVisionImage . fromBitmap ( bitmap )

detector.detectInImage(firebaseImage) .addOnSuccessListener { processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result) }

private fun doOcrDetection(bitmap: Bitmap){ val detector = FirebaseVision.getInstance() .visionTextDetector val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap) detector . detectInImage ( firebaseImage )

. addOnSuccessListener {

processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result) }

}

. addOnFailureListener {

Toast.makeText(/../“Error detecting Text $it”/../)

}

}

private fun doOcrDetection(bitmap: Bitmap){ val detector = FirebaseVision.getInstance() .visionTextDetector val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap) detector . detectInImage ( firebaseImage )

. addOnSuccessListener {

        processedBitmap

. postValue ( ocrProcessor . drawBoxes ( bitmap , it ))

var result

String ()

        it

. blocks . forEach {

            result

+=

" "

it . text textResult . postValue ( result )

}

}

. addOnFailureListener {

Toast . makeText ( /../ “Error detecting Text $it” /../ )

}

}

Custom Tensorflow Models . / Works offline

TensorFlow

Retrain existing model 0

mobilenet_v 1

What if I could tell what kind of chips I was eating?

Gather
Training
Data FFMPEG Folders of Images Retrain with
new images Optimize
for mobile Embed in app Store in Firebase App
uses
model

Gather training
data

Export to Images using ffmpeg ffmpeg -i flings.mp4 flings/flings_%04d.jpg

Folders of images

Retrain with new images python -m scripts.retrain

--bottleneck_dir=tf_files/bottlenecks
--how_many_training_steps=500
--model_dir=tf_files/models/
--summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}"
--output_graph=tf_files/ retrained_graph.pb

--output_labels=tf_files/retrained_labels.txt
--architecture="${ARCHITECTURE}"
--image_dir=training_data/ south_african_chips

Optimize for mobile bazel-bin/tensorflow/contrib/lite/toco/toco
--input_file=AgencyDay/ retrained_graph.pb

--output_file=AgencyDay/ chips_optimized_graph.tflite

--input_format=TENSORFLOW_GRAPHDEF
--output_format=TFLITE
--input_shape=1,${IMAGE_SIZE},${IMAGE_SIZE},3
--input_array=input
--output_array=final_result
--inference_type=FLOAT
--input_data_type=FLOAT

Insert into App https://codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/

https://github.com/googlecodelabs/tensorflow-for-poets-2

Nik Nak 1

Or Not 2

bit.ly/mlkit-riggaroo

What if our model changes?

Ship an app update 3

And hope that people download it 4

Host on Firebase "

Updates automatically downloaded

val cloudSource

FirebaseCloudModelSource . Builder ( "my_cloud_model" )

. enableModelUpdates ( true )

. setInitialDownloadConditions ( conditions )

. setUpdatesDownloadConditions ( conditions )

. build () FirebaseModelManager . getInstance () . registerCloudModelSource ( cloudSource ) ……

g.co/codelabs/mlkit-android-custom-model

You don’t need to be a ML Expert to take advantage of ML in your apps! !

Thank you!

Resources

https://codelabs.developers.google.com/ codelabs/tensorflow-for-poets

https://codelabs.developers.google.com/ codelabs/tensorflow-for-poets-2-tflite/

https://codelabs.developers.google.com/ codelabs/mlkit-android-custom-model/ #0

https://github.com/riggaroo/android- demo-mlkit