Slide 1

Android Developers Guide to Machine Learning ! With MLKit, Tensorflow & Firebase "

Slide 2

Rebecca Franks @riggaroo

Google Developer Expert
Android @ Over Pluralsight Author GDG Johannesburg Organiser

Slide 3

Who considers themselves an expert in Machine Learning?

Slide 4

Slide 5

What is Machine Learning?

Slide 6

Machine learning is an application of Artificial Intelligence in which we input a lot of data and let the machines learn

“by themselves”

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15

Face detection $ % Works offline

Slide 16

Slide 17

Slide 18

Slide 19

Slide 20

Slide 21

Slide 22

Demo

Slide 23

Slide 24

& ⛩

Landmark detection ( )

Slide 25

Slide 26

Slide 27

Demo

Slide 28

Slide 29

Image Labelling * Works offline

Slide 30

Slide 31

Barcode scanning + Works offline

Slide 32

Slide 33

Slide 34

val options

FirebaseVisionBarcodeDetectorOptions . Builder ()

. setBarcodeFormats ( FirebaseVisionBarcode . FORMAT_QR_CODE )

. build ()

val image = FirebaseVisionImage.fromBitmap(bitmap) val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector.detectInImage(image) .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it)) var result = String()

Slide 35

val options = FirebaseVisionBarcodeDetectorOptions.Builder() .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_QR_CODE) .build() val image

FirebaseVisionImage . fromBitmap ( bitmap )

val detector

FirebaseVision . getInstance () . getVisionBarcodeDetector ( options )

detector.detectInImage(image) .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it)) var result = String() it.forEach { result += "VALUE TYPE: ${it.valueType} Raw Value: ${it.rawValue}" textResult.postValue(result) }

Slide 36

val image = FirebaseVisionImage.fromBitmap(bitmap) val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector . detectInImage ( image )

. addOnSuccessListener {

    processedBitmap

. postValue ( barcodeProcessor . drawBoxes ( bitmap , it ))

var result

String ()

it

. forEach {

        result

"VALUE TYPE: ${it.valueType} Raw Value: ${it.rawValue}"

        textResult

. postValue ( result )

}

}. addOnFailureListener {

    textResult

. postValue ( it . message )

}

Slide 37

Slide 38

OCR ,

Works offline

Slide 39

Slide 40

On Device vs Cloud

Slide 41

private fun doOcrDetection ( bitmap :

Bitmap ){

val detector

FirebaseVision . getInstance ()

. visionTextDetector val firebaseImage

FirebaseVisionImage . fromBitmap ( bitmap )

detector.detectInImage(firebaseImage) .addOnSuccessListener { processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result) }

Slide 42

private fun doOcrDetection(bitmap: Bitmap){ val detector = FirebaseVision.getInstance() .visionTextDetector val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap) detector . detectInImage ( firebaseImage )

. addOnSuccessListener {

processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result) }

}

. addOnFailureListener {

Toast.makeText(/../“Error detecting Text $it”/../)

}

Slide 43

. addOnSuccessListener {

        processedBitmap

. postValue ( ocrProcessor . drawBoxes ( bitmap , it ))

var result

String ()

it

. blocks . forEach {

            result

" "

it . text textResult . postValue ( result )

}

. addOnFailureListener {

Toast . makeText ( /../ “Error detecting Text $it” /../ )

}

Slide 44

Slide 45

Custom Tensorflow Models . / Works offline

Slide 46

TensorFlow

Slide 47

Slide 48

Slide 49

Retrain existing model 0

mobilenet_v 1

Slide 50

What if I could tell what kind of chips I was eating?

Slide 51

Gather
Training
Data FFMPEG Folders of Images Retrain with
new images Optimize
for mobile Embed in app Store in Firebase App
uses
model

Slide 52

Gather training
data

Slide 53

Export to Images using ffmpeg ffmpeg -i flings.mp4 flings/flings_%04d.jpg

Slide 54

Folders of images

Slide 55

Retrain with new images python -m scripts.retrain

--bottleneck_dir=tf_files/bottlenecks
--how_many_training_steps=500
--model_dir=tf_files/models/
--summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}"
--output_graph=tf_files/ retrained_graph.pb

--output_labels=tf_files/retrained_labels.txt
--architecture="${ARCHITECTURE}"
--image_dir=training_data/ south_african_chips

Slide 56

Optimize for mobile bazel-bin/tensorflow/contrib/lite/toco/toco
--input_file=AgencyDay/ retrained_graph.pb

--output_file=AgencyDay/ chips_optimized_graph.tflite

--input_format=TENSORFLOW_GRAPHDEF
--output_format=TFLITE
--input_shape=1,${IMAGE_SIZE},${IMAGE_SIZE},3
--input_array=input
--output_array=final_result
--inference_type=FLOAT
--input_data_type=FLOAT