SlideShare a Scribd company logo

Building a custom machine learning model on android

•
•
Isabel Palomar
Isabel Palomar

This document provides an overview of building a custom machine learning model for image classification on Android. It begins with discussing challenges and ideas, then covers key deep learning concepts like data, tasks, models, loss functions, learning algorithms and evaluation. It explains that a MobileNet model will be retrained for classifying images of artisanal beers. The document also discusses converting the model to TensorFlow Lite and implementing image classification in an Android app using the camera and a TensorFlow Lite interpreter to get classification results.

Related slideshows

Start machine learning in 5 simple steps
Start machine learning in 5 simple stepsStart machine learning in 5 simple steps
Start machine learning in 5 simple steps
 
AIML4 CNN lab256 1hr (111-1).pdf
AIML4 CNN lab256 1hr (111-1).pdfAIML4 CNN lab256 1hr (111-1).pdf
AIML4 CNN lab256 1hr (111-1).pdf
 
MLFlow: Platform for Complete Machine Learning Lifecycle
MLFlow: Platform for Complete Machine Learning Lifecycle MLFlow: Platform for Complete Machine Learning Lifecycle
MLFlow: Platform for Complete Machine Learning Lifecycle
 
1 of 56
Download to read offline
Building a Custom Machine Learning Model on Android
Hello! I am Isabel Palomar 2
Agenda ● Challenges and Initial ideas ● Main Deep Learning concepts ● Ok, but, what about Android? … that’s all 3 ��
1. CHALLENGES AND INITIAL IDEAS Getting started with Machine Learning and Android
Production-ready for common use cases 5
I want to create a custom model HOW? 6
“ 77 @jeremyphoward @math_rachel I started a Deep Learning Course 8 months ago at the University of San Francisco. 💃 MY STORY: TAKING MY FIRST DEEP LEARNING CLASS
8 🤯 After the class….. The key outcome of this lesson is that we'll have trained an image classifier which can recognize pet breeds at state of the art accuracy. The key to this success is the use of transfer learning, which will be a key platform for much of this course. We also discuss how to set the most important hyper-parameter when training neural networks: the learning rate, using Leslie Smith's fantastic learning rate finder method. Finally, we'll look at the important but rarely discussed topic of labeling, and learn about some of the features that fastai provides for allowing you to easily add labels to your images. https://course.fast.ai/videos/?lesson=1
challenges…. ‐ Many courses, even basic, assume that you already know the subject. ‐ Reaching the final result without learning the basics is not good. 9
“When you are starting to learn about Deep Learning it seems that there are thousands of concepts, mathematical functions and scientific articles that you have to read. 10 myths
2. Main DEEP LEARNING CONCEPTS Let’s understand the general concepts first
MACHINE LEARNING CONCEPTS Data Task Model 12 Loss Function Learning Algorithm Evaluation
Let’s take a look of the implementation We are going to build an app to classify the artisanal beers of Cervecería Colima Place your screenshot here 13
1.- dATA Data is distinct pieces of information which acts as a fuel 14
Formats Data can come in different formats depending on what you want to solve 15
DATA FOR OUR EXAMPLE 16
How? Where do we get data from? Data curation is the organization and integration of data collected from various sources. 17 Techniques You can use techniques like Questionnaires and surveys, conducting interviews, using data scraping and data crawling techniques.
Public datasets ● Google AI ● UCI ML Repository ● Data.gov.in ● Kaggle Where do we get data from? Crowdsourcing Marketplaces ● Amazon Mechanical Turk ● Dataturks ● Figure-eight 18
BACK TO OUR EXAMPLE... ● Google Images ● https://github.com/hardikvasa/google-images-download ● https://forums.fast.ai/t/tips-for-building-large-image-datasets/26688 19
2.- task The problem to resolve 20
Examples of tasks 21
TASK FOR OUR EXAMPLE 22 Classify Images of Artisanal Beers
Image classification A common use of machine learning is to identify what an image represents. The task of predicting what an image represents is called image classification. 23
3.- model The mathematical formulation of a task. 24
models 25 There are many models that are created over the years. Each model has its own advantages and disadvantages based on the type of data on which we are creating a model.
IMAGE CLASSIFICATION MODEL An image classification model is trained to recognize various classes of images. 26 When we subsequently provide a new image as input to the model, it will output the probabilities of the image representing each of the types it was trained on.
An example output might be as follows: Beer type Probability Cayaco 0.02 Colimita 0.96 Piedra Lisa 0.01 Ticus 0.00 Paramo 0.01 27 Based on the output, we can see that the classification model has predicted that the image has a high probability of representing a Colimita Beer.
In this example, we will retrain a MobileNet. MobileNet is a a small efficient convolutional neural network. https://ai.googleblog.com/2017/06/mobilenets-open-source-models-for.html Model for our example 28
Retraining the mobileNet model 29 We use MobileNet model and retrain it. python3 -m scripts.retrain --bottleneck_dir=tf_files/bottlenecks --model_dir=tf_files/models/"${ARCHITECTURE}" --summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}" --output_graph=tf_files/retrained_graph.pb --output_labels=tf_files/retrained_labels.txt --architecture="${ARCHITECTURE}" --image_dir=tf_files/beer_photos IMAGE_SIZE=224 ARCHITECTURE="mobilenet_0.50_${IMAGE_SIZE}" tHE RESULT...
USING THE RETRAINED MODEL 3030 Evaluation time (1-image): 0.250s ticus (score=0.99956) paramo (score=0.00043) cayaco (score=0.00000) piedra lisa (score=0.00000) colimita (score=0.00000) python3 -m scripts.label_image --graph=tf_files/retrained_graph.pb --image=tf_files/beer_photos/ticus/"3. ticus.jpg"
4.- loss function How do we know which model is better? Loss function (also known as the error) answers this question. 31
Models and loss function How good a prediction model does in terms of being able to predict the expected outcome. 32
Classification losses: ● Mean Square Error/L2 Loss ● Mean Absolute Error/L1 Loss Regression losses: ● Hinge Loss/Multi-class SVM Loss ● Cross Entropy ● Loss/Negative Log Likelihood LOSS FUNCTIONS To know which model is good for our data, we compute the loss function by comparing the predicted outputs to actual output. 33
5.- learning algorithm The Learning Algorithms also known as Optimization algorithms helps us to minimize Error 34
First Order Optimization Algorithms  ● Gradient Descent Types of learning algorithms Second Order Optimization Algorithms ● Hessian https://towardsdatascience.com/types-of-optimization-algorithms-used-in-neural-networks-and- ways-to-optimize-gradient-95ae5d39529f 35
Is something you do everyday... You are optimizing variables and basing your personal decisions all day long, most of the time without even recognizing the process consciously https://mitsloan.mit.edu/ideas-made-to-matter/how-to-use -algorithms-to-solve-everyday-problems 36
6.- evaluation To compute a score for our ML model, we need to evaluate it. 37
evaluation There are many performance metrics to achieve this. One simple such metric is Accuracy. 38
Evaluation/performance metrics 39 ● Confusion matrix ● Area Under the ROC curve (AUC — ROC) ● Root Mean Squared Error (RMSE) (used in regression)
DEEP LEARNING CONCEPTS Data Task Model 40 Loss Function Learning Algorithm Evaluation
3. OKAY, BUT WHAT ABOUT ANDROID? How to use our model in my app
MACHINE LEARNING IN YOUR APPS ● ML Kit For Firebase ● Core ML (Apple) ● TensorFlow Lite ● Cloud-based web services ● Your own service Place your screenshot here 42
TENSORFLOW LITE TensorFlow Lite is an open source deep learning framework for on-device inference. 43
USING THE RETRAINED MODEL 4444 Evaluation time (1-image): 0.250s ticus (score=0.99956) paramo (score=0.00043) cayaco (score=0.00000) piedra lisa (score=0.00000) colimita (score=0.00000) python3 -m scripts.label_image --graph=tf_files/retrained_graph.pb --image=tf_files/beer_photos/ticus/"3. ticus.jpg"
TENSORFLOW LITE 45 TensorFlow Lite is a set of tools to help developers run TensorFlow models on mobile, embedded, and IoT devices. ● TensorFlow Lite converter ● TensorFlow Lite interpreter TensorFlow Lite converter Converts TensorFlow models into an efficient form for use by the interpreter
Command line: tflite_convert Starting from TensorFlow 1.9, the command-line tool tflite_convert is installed as part of the Python package. 46 pip install --upgrade "tensorflow==1.9.*"
Using TFLite Converter 47 Output:IMAGE_SIZE=224 tflite_convert --graph_def_file=tf_files/retrained_graph.pb --output_file=tf_files/optimized_graph.lite --input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE --input_shape=1,${IMAGE_SIZE},${IMAGE_SIZE},3 --input_array=input --output_array=final_result --inference_type=FLOAT --input_data_type=FLOAT
recap 48 ? @riggaroo
Required files 49 Optimized graph cp tf_files/optimized_graph.lite android/tflite/app/src/main/assets/graph.lite cp tf_files/retrained_labels.txt android/tflite/app/src/main/assets/labels.txt Labels
repositories { maven { url 'https://google.bintray.com/tensorflow' } } dependencies { // ... compile 'org.tensorflow:tensorflow-lite:+' } TensorFlow Lite interpreter 50 android { aaptOptions { noCompress "tflite" noCompress "lite" } } The TensorFlow Lite interpreter is designed to be lean and fast. The interpreter uses a static graph ordering and a custom (less-dynamic) memory allocator to ensure minimal load, initialization, and execution latency. dependencies settings
Load model and create interpreter protected Classifier… { tfliteOptions.setNumThreads(numThreads); tflite = new Interpreter(tfliteModel, tfliteOptions); labels = loadLabelList(activity); ... } 51 // Name of the model file stored in Assets. private static final String MODEL_PATH = "graph.lite"; // Name of the label file stored in Assets. private static final String LABEL_PATH = "labels.txt";
cAMERA, Read the labels….. 52 https://developer.android.com/training/camerax // Convert the image to bytes convertBitmapToByteBuffer(bitmap); // An array to hold inference results, to be feed into Tensorflow Lite as outputs. PriorityQueue<Map.Entry<String, Float>> sortedLabels = new PriorityQueue<>( RESULTS_TO_SHOW, (element1, element2) -> (element1.getValue()).compareTo(element2.getValue()));
Show the results 53 // Get the results textToShow = String.format("n%s: %4.2f", label.getKey(), label.getValue()) // Label (In this case PARAMO) label.getKey() // Value (In this case 1.0) label.getValue() ticus (score=0.00000) paramo (score=1.00000) cayaco (score=0.00000) piedra lisa (score=0.00000) colimita (score=0.00000)
We did it! 54 Place your screenshot here @riggaroo
Call to action! Now you are ready to start building your first custom ML model. What are you going to build? 55
56 THANKS! You can find me at @isabelpalomar

More Related Content

Building a custom machine learning model on android

  • 1. Building a Custom Machine Learning Model on Android
  • 2. Hello! I am Isabel Palomar 2
  • 3. Agenda ● Challenges and Initial ideas ● Main Deep Learning concepts ● Ok, but, what about Android? … that’s all 3 ��
  • 4. 1. CHALLENGES AND INITIAL IDEAS Getting started with Machine Learning and Android
  • 6. I want to create a custom model HOW? 6
  • 7. “ 77 @jeremyphoward @math_rachel I started a Deep Learning Course 8 months ago at the University of San Francisco. 💃 MY STORY: TAKING MY FIRST DEEP LEARNING CLASS
  • 8. 8 🤯 After the class….. The key outcome of this lesson is that we'll have trained an image classifier which can recognize pet breeds at state of the art accuracy. The key to this success is the use of transfer learning, which will be a key platform for much of this course. We also discuss how to set the most important hyper-parameter when training neural networks: the learning rate, using Leslie Smith's fantastic learning rate finder method. Finally, we'll look at the important but rarely discussed topic of labeling, and learn about some of the features that fastai provides for allowing you to easily add labels to your images. https://course.fast.ai/videos/?lesson=1
  • 9. challenges…. ‐ Many courses, even basic, assume that you already know the subject. ‐ Reaching the final result without learning the basics is not good. 9
  • 10. “When you are starting to learn about Deep Learning it seems that there are thousands of concepts, mathematical functions and scientific articles that you have to read. 10 myths
  • 11. 2. Main DEEP LEARNING CONCEPTS Let’s understand the general concepts first
  • 12. MACHINE LEARNING CONCEPTS Data Task Model 12 Loss Function Learning Algorithm Evaluation
  • 13. Let’s take a look of the implementation We are going to build an app to classify the artisanal beers of Cervecería Colima Place your screenshot here 13
  • 14. 1.- dATA Data is distinct pieces of information which acts as a fuel 14
  • 15. Formats Data can come in different formats depending on what you want to solve 15
  • 16. DATA FOR OUR EXAMPLE 16
  • 17. How? Where do we get data from? Data curation is the organization and integration of data collected from various sources. 17 Techniques You can use techniques like Questionnaires and surveys, conducting interviews, using data scraping and data crawling techniques.
  • 18. Public datasets ● Google AI ● UCI ML Repository ● Data.gov.in ● Kaggle Where do we get data from? Crowdsourcing Marketplaces ● Amazon Mechanical Turk ● Dataturks ● Figure-eight 18
  • 19. BACK TO OUR EXAMPLE... ● Google Images ● https://github.com/hardikvasa/google-images-download ● https://forums.fast.ai/t/tips-for-building-large-image-datasets/26688 19
  • 20. 2.- task The problem to resolve 20
  • 22. TASK FOR OUR EXAMPLE 22 Classify Images of Artisanal Beers
  • 23. Image classification A common use of machine learning is to identify what an image represents. The task of predicting what an image represents is called image classification. 23
  • 24. 3.- model The mathematical formulation of a task. 24
  • 25. models 25 There are many models that are created over the years. Each model has its own advantages and disadvantages based on the type of data on which we are creating a model.
  • 26. IMAGE CLASSIFICATION MODEL An image classification model is trained to recognize various classes of images. 26 When we subsequently provide a new image as input to the model, it will output the probabilities of the image representing each of the types it was trained on.
  • 27. An example output might be as follows: Beer type Probability Cayaco 0.02 Colimita 0.96 Piedra Lisa 0.01 Ticus 0.00 Paramo 0.01 27 Based on the output, we can see that the classification model has predicted that the image has a high probability of representing a Colimita Beer.
  • 28. In this example, we will retrain a MobileNet. MobileNet is a a small efficient convolutional neural network. https://ai.googleblog.com/2017/06/mobilenets-open-source-models-for.html Model for our example 28
  • 29. Retraining the mobileNet model 29 We use MobileNet model and retrain it. python3 -m scripts.retrain --bottleneck_dir=tf_files/bottlenecks --model_dir=tf_files/models/"${ARCHITECTURE}" --summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}" --output_graph=tf_files/retrained_graph.pb --output_labels=tf_files/retrained_labels.txt --architecture="${ARCHITECTURE}" --image_dir=tf_files/beer_photos IMAGE_SIZE=224 ARCHITECTURE="mobilenet_0.50_${IMAGE_SIZE}" tHE RESULT...
  • 30. USING THE RETRAINED MODEL 3030 Evaluation time (1-image): 0.250s ticus (score=0.99956) paramo (score=0.00043) cayaco (score=0.00000) piedra lisa (score=0.00000) colimita (score=0.00000) python3 -m scripts.label_image --graph=tf_files/retrained_graph.pb --image=tf_files/beer_photos/ticus/"3. ticus.jpg"
  • 31. 4.- loss function How do we know which model is better? Loss function (also known as the error) answers this question. 31
  • 32. Models and loss function How good a prediction model does in terms of being able to predict the expected outcome. 32
  • 33. Classification losses: ● Mean Square Error/L2 Loss ● Mean Absolute Error/L1 Loss Regression losses: ● Hinge Loss/Multi-class SVM Loss ● Cross Entropy ● Loss/Negative Log Likelihood LOSS FUNCTIONS To know which model is good for our data, we compute the loss function by comparing the predicted outputs to actual output. 33
  • 34. 5.- learning algorithm The Learning Algorithms also known as Optimization algorithms helps us to minimize Error 34
  • 35. First Order Optimization Algorithms  ● Gradient Descent Types of learning algorithms Second Order Optimization Algorithms ● Hessian https://towardsdatascience.com/types-of-optimization-algorithms-used-in-neural-networks-and- ways-to-optimize-gradient-95ae5d39529f 35
  • 36. Is something you do everyday... You are optimizing variables and basing your personal decisions all day long, most of the time without even recognizing the process consciously https://mitsloan.mit.edu/ideas-made-to-matter/how-to-use -algorithms-to-solve-everyday-problems 36
  • 37. 6.- evaluation To compute a score for our ML model, we need to evaluate it. 37
  • 38. evaluation There are many performance metrics to achieve this. One simple such metric is Accuracy. 38
  • 39. Evaluation/performance metrics 39 ● Confusion matrix ● Area Under the ROC curve (AUC — ROC) ● Root Mean Squared Error (RMSE) (used in regression)
  • 40. DEEP LEARNING CONCEPTS Data Task Model 40 Loss Function Learning Algorithm Evaluation
  • 41. 3. OKAY, BUT WHAT ABOUT ANDROID? How to use our model in my app
  • 42. MACHINE LEARNING IN YOUR APPS ● ML Kit For Firebase ● Core ML (Apple) ● TensorFlow Lite ● Cloud-based web services ● Your own service Place your screenshot here 42
  • 43. TENSORFLOW LITE TensorFlow Lite is an open source deep learning framework for on-device inference. 43
  • 44. USING THE RETRAINED MODEL 4444 Evaluation time (1-image): 0.250s ticus (score=0.99956) paramo (score=0.00043) cayaco (score=0.00000) piedra lisa (score=0.00000) colimita (score=0.00000) python3 -m scripts.label_image --graph=tf_files/retrained_graph.pb --image=tf_files/beer_photos/ticus/"3. ticus.jpg"
  • 45. TENSORFLOW LITE 45 TensorFlow Lite is a set of tools to help developers run TensorFlow models on mobile, embedded, and IoT devices. ● TensorFlow Lite converter ● TensorFlow Lite interpreter TensorFlow Lite converter Converts TensorFlow models into an efficient form for use by the interpreter
  • 46. Command line: tflite_convert Starting from TensorFlow 1.9, the command-line tool tflite_convert is installed as part of the Python package. 46 pip install --upgrade "tensorflow==1.9.*"
  • 47. Using TFLite Converter 47 Output:IMAGE_SIZE=224 tflite_convert --graph_def_file=tf_files/retrained_graph.pb --output_file=tf_files/optimized_graph.lite --input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE --input_shape=1,${IMAGE_SIZE},${IMAGE_SIZE},3 --input_array=input --output_array=final_result --inference_type=FLOAT --input_data_type=FLOAT
  • 49. Required files 49 Optimized graph cp tf_files/optimized_graph.lite android/tflite/app/src/main/assets/graph.lite cp tf_files/retrained_labels.txt android/tflite/app/src/main/assets/labels.txt Labels
  • 50. repositories { maven { url 'https://google.bintray.com/tensorflow' } } dependencies { // ... compile 'org.tensorflow:tensorflow-lite:+' } TensorFlow Lite interpreter 50 android { aaptOptions { noCompress "tflite" noCompress "lite" } } The TensorFlow Lite interpreter is designed to be lean and fast. The interpreter uses a static graph ordering and a custom (less-dynamic) memory allocator to ensure minimal load, initialization, and execution latency. dependencies settings
  • 51. Load model and create interpreter protected Classifier… { tfliteOptions.setNumThreads(numThreads); tflite = new Interpreter(tfliteModel, tfliteOptions); labels = loadLabelList(activity); ... } 51 // Name of the model file stored in Assets. private static final String MODEL_PATH = "graph.lite"; // Name of the label file stored in Assets. private static final String LABEL_PATH = "labels.txt";
  • 52. cAMERA, Read the labels….. 52 https://developer.android.com/training/camerax // Convert the image to bytes convertBitmapToByteBuffer(bitmap); // An array to hold inference results, to be feed into Tensorflow Lite as outputs. PriorityQueue<Map.Entry<String, Float>> sortedLabels = new PriorityQueue<>( RESULTS_TO_SHOW, (element1, element2) -> (element1.getValue()).compareTo(element2.getValue()));
  • 53. Show the results 53 // Get the results textToShow = String.format("n%s: %4.2f", label.getKey(), label.getValue()) // Label (In this case PARAMO) label.getKey() // Value (In this case 1.0) label.getValue() ticus (score=0.00000) paramo (score=1.00000) cayaco (score=0.00000) piedra lisa (score=0.00000) colimita (score=0.00000)
  • 54. We did it! 54 Place your screenshot here @riggaroo
  • 55. Call to action! Now you are ready to start building your first custom ML model. What are you going to build? 55
  • 56. 56 THANKS! You can find me at @isabelpalomar