Mlkit android

Mlkit android DEFAULT

Using ML Kit in Android


It has been a little more than a year since Google announced that they are changing their core strategy from “Mobile First” to “AI first”. Since then a lot of focus in technology is shifted to Artificial Intelligence and quite visible efforts are being put in this area from all the dominant players of web — Google, Amazon and Facebook. As a result we are witnessing an advent of a lot of services, frameworks or platforms coming out for developers. The interest in Machine Learning and Deep learning has been increased by 50 times in last few years. Albeit an interesting fact is that Deep Learning or Machine Learning is not a new thing, it's there for a couple of decades but only now we have become capable of using and deploying ML in terms of computing and processing powers.

Birth of Tensor Flow aka ML for all:

Google invested a lot in this direction and integrated ML into many of their primary products, then they went ahead and put their efforts into public usage to help outside developers. Google released TensorFlow for developers to create, train and serve their custom ML models. Facebook has their very own PyTorch framework and Amazon is also providing ML and AI services via AWS. Machine Learning is a pretty vast and complex area and it is categorized as “doable but hard” as ML needs expertise in Maths, statistics and Data Science. Emerging startup can not always have good ML/AI talent in their small teams. To integrate ML into their app in quick fashion and to use ML in their use-cases, there are some ready made solutions built for them. TensorFlow Lite and ML Kit is one such solution. The focus of this article is ML Kit.

ML Kit

So what is ML Kit? As the name says it is a Machine Learning kit for developers and provides a lot of functionality including but not limited to image analysis. It all started from an offering from Google called “Google Vision” 2 years ago. Google Vision was an Image Analysis framework which aimed to help developers in detecting objects, faces and texts in a given image. Although Google Vision was a cloud hosted solution, there was a mobile peer also available with name “Mobile Vision” which was focused on Android app developers India to use the library for face detection, face landmark detection, barcode reading and text detection. Since the release of this library there are a lot of advancements in the field of ML and AI. The new efforts gave birth to TensorFlow and also changed the mechanism of how images are analyzed. Mobile Vision was deprecated and replaced by ML Kit. So the objective of ML Kit is same as Mobile Vision but it is more advanced and capable of a lot more things in better way. ML Kit is available under Firebase suite of features. Firebase is being extensively used by Android/iOS developers for authentication, real-time database, analytics, crash reporting and AB testing.

Let’s explore ML Kit now

ML Kit provides following features out of the box:

  • Text recognition
  • Face detection
  • Image labelling
  • Barcode Scanning
  • Recognising landmarks

As I mentioned earlier, it is hard to implement ML in apps from scratch, development teams may not have ML experts in team so ML kit resolves this issue. Using ML kit, developers can take advantage of these features without having to do anything from scratch. ML kit provides REST API’s to do the image recognition and above tasks. And also it is not limited to this. If developers want to go deeper and have enough understanding of how to create, train and deploy ML models they can use ML Kit to do that on Firebase server. There is also “TensorFlow Lite” version of TF models which are very light weight and give almost the same kind of accuracy.


On device ML execution

One more pretty interesting thing about ML Kit is that they have also focused on ‘on device’ ML execution. On device execution, using this we can do the image detection kind of things on the device without needing to contact the server. The ML framework will download the model on the device or you can ship it with your APK and it will execute using the given input and return the result. The obvious advantages here are low latency, no network need (offline) and no cost. It’s free. The downside here is that it may not be as accurate as the Cloud API but developers can choose any of this on the basis of their use case.

Integrating and using ML Kit in Android app

Since ML Kit is part of Firebase, first of all Firebase is needed to be setup in project. It can be done manually or using Android Studio Firebase Tool. If Firebase tool is not available in your Android Studio go to this link to read how to install the Tool. It is very helpful if you are setting up Firebase for the first time. I will quickly go through the steps of Firebase setup. You can also get the source of sample project from the github here.


Not too long ago, machine learning (ML) used to sound like a magic ?. Details on what, and how, it should be used were quite vague for many of us for some time.

That was then. Fast forward to now, and one of the most exciting announcements of Google I/O 2018, for me, was MLKit. Briefly, MLKit takes some common ML use cases, and wraps them up with a nice API.

In addition to this, if your app handles a specific use case, MLKit allows you to create and use custom models, along with useful version management features.

One of the out-of-the-box use cases is image labelling, also known as image classification. This means taking an image and detecting the entities it contains, such as: animals, fruits, activities and so on. Each image input will get an output of a list of entities (labels) with a score that represents the level of confidence that this entity is indeed present in the image. These labels can be used in order to perform actions such as content moderation, filtering or search.

Also, because mobile devices have limited resources and minimizing data consumption is often a blessing, working with metadata rather than an entire photo can help with matters of performance, privacy, bandwidth, offline support and more. For example, on a chat app, being able to send only labels and not an entire photo, can benefit a lot.

This tutorial will guide you through writing a mobile app that can take an image and detect the entities it contains. It has 3 parts:

Hands down, my favourite food in the world is strawberries ?! I can eat them everyday, all day, and it would make me so happy!

Let’s create an app that will take an image, and then detect if in contains a strawberry or not!

The app you will create:

The user selects an image, and a button to choose how to classify it.

The UI passes the image ( object) to class, which will send the image to a specific Classifier that was selected (, or . This tutorial will only cover the first two). Each Classifier will process the input, run the model, process the output if needed, and then send the result to that will prepare it to make it as easy as possible for the UI to display.

Before getting started:

  1. Clone this project with code to get started, and the implementation per step

  2. Add Firebase to your app:

  • Under General tab → under Your Apps section, choose 'add app'.

  • Follow the steps in the Firebase tutorial, to add Firebase to your app.

  1. Add library to your app: on your app-level file add:

As mentioned, this tutorial will cover running both local and cloud based detectors. Each has 4 steps:

  1. Setting up (it’s not cheating :) doesn’t really count as a step…)
  2. Setup the Classifier
  3. Process the input
  4. Process the output

Note: If you prefer to follow long with the final code, you can find it on branch 1.run_local_model of the demo's repo.

Running a local (on-device) model

Choosing a local model is the lightweight, offline supported option, and it's free. In return, it has 400+ labels so the accuracy is limited, which we must take into account.

The UI takes the bitmap → calls → calls

If you prefer to follow along with the final code, get it on branch 1.run_local_model.

Step 0: Setting up

  1. Adding to your app the local detector, facilitated by Firebase MLKit:

On your app-level file add:

Optional, but recommended: by default, the ML model itself will be downloaded only once you execute the detector. It means that there will be some latency at the first execution, as well as network access required. To bypass that, and have the ML model downloaded as the app is installed from Play Store, simply add the following declaration to your app's AndroidManifest.xml file:

Step 1: Setup the Classifier

Create class that holds the detector object:

This is the basic detector instance. You can be more picky about the output returned, and add Confidence Threshold, which is between 0–1, with 0.5 as a default.

Step 2: Process The Input

knows how to work with an input of type . You can obtain a instance from either:

(this is what we'll use here) , ,  (from media, for example the device camera), , or .

Processing a is done like this:

Tip: One of the reasons we'd want to use a local model is because the execution is quicker, however, executing any model takes some time. If you use the model on a real-time application, you might need the results even faster. Reducing the bitmap size before moving to the next step, can improve the model's processing time.

Step 3: Run The Model

This is where the magic happens! ? Since the model does take some computation time, we should have the model run asynchronously and return the success or failure result using listeners.

Step 4: Process The Output

The detection output is provided on . I prefer to have the passed to from , that handles the communication between the UI and .

The UI calls , which should look something like that:

On :

just prepares the output labels to display in the UI.

In my specific case, I chose to display the 3 results with highest probability. You may choose any other format type. To complete the picture, this is my implementation:


is a simple interface I created, in order to communicate the results back to the UI display. We could have also used any other other methods available to do this. It's a matter of preference.

That’s it!

You used your first ML model to classify an image! ? How simple was that?!

Let's run the app and see some results! Get the final code for this part on this demo’s repo , on branch 1.run_local_model

Pretty good! We got some general labels like 'food' or 'fruit', that definitely fit the image. For some apps use cases this model fits perfectly fine. It can help group images, perform a search and so on. But for our case, we expect a model that can specify which fruit is in the photo.

Let’s try to get some more indicative and accurate labels, by using the cloud based detector, which has 10,000|+ labels available:

Running a cloud based model

Step 0: Setting up

MLKit's Cloud based models belong to the Cloud Vision API, which you have to make sure is enabled for your project:

  1. Using a cloud-based model requires payment over a quota of 1000+ monthly uses. For demo and development purposes, it’s not likely that you’ll get near that quota. However, you must upgrade your Firebase project plan, so that theoretically it can be charged if needed. Upgrade your Spark plan project, which is free, to a Blaze plan, which is pay as you go, and enables you to use the Cloud Vision APIs. You can do so in the Firebase console.

  2. Enable the Cloud Vision API, on the Cloud Console API library. On the top menu, select your Firebase project, and if not already enabled, click Enable.

Note: For development, this configuration will do. However, prior to deploying to production, you should take some extra steps to ensure that no unauthorised calls are being made with your account. For that case, check out the instructions here.

Step 1: Setup The Classifier

Create a class that holds the detector object:

This is almost the same as the above , except the type of the detector.

There are a few extra options we can set on the detector:

  •  — by default 10 results will return. If you need more than that, you'd have to specify it. On the other end, when designing the demo app I decided to only present the top 3 results. I can define it here and make the computation a little faster.

  •  — can be either STABLE_MODEL or LATEST_MODEL, the latter is default.

Step 2: Process The Input

Similarly to , uses an input of , which we will obtain it from a , to facilitate the UI;

Step 3: Run The Model

As the previous steps, this step is incredibly similar the what we did to run the local model:

Step 4: Process The Output

As the local model is different than the cloud based model, their outputs will be different, so that the object type we get as the response on is different per detector. Yet, the objects are quite the same to work with.

On :

Once again, processing the results for the UI to present is down to your own decision on what the UI presents. For this example:

That’s pretty much it! ?

Let's see some results: The code for this post can be found on the repo, on branch 2.run_cloud_model

As expected, the model took a little longer, but can now tell which specific fruit is in the image. Also, it is more than 90% confident of the result, comparing to 70–80% confidence for the local model.

I hope this helps you to understand how simple and fun it is to use Firebase MLKit. Using the other models: face detection, barcode scanning, etc.. works in a very similar way and I encourage you to try it out!

Can we get even better results? We’ll explore that using a custom model as well, on an upcoming post.

What’s Next?

If you want to learn more about what machine learning is and how does it work, check out these developer friendly intro blog posts:,,

For more information on why to use MLKit see this post and the official documentation

  1. Clarinet sheet music
  2. Embroidery library designs
  3. Erect meaning
  4. El paso police scanner frequency
  5. Todoroki car decal

Google’s ML Kit makes it easy to add AI smarts to iOS and Android apps

At its I/O developer conference, Google today introduced ML Kit, a new software development kit (SDK) for app developers on iOS and Android that allows them to integrate into their apps a number of pre-built Google-provided machine learning models. One nifty twist here is that these models — which support text recognition, face detection, barcode scanning, image labeling and landmark recognition — are available both online and offline, depending on network availability and the developer’s preference.

In the coming months, Google plans to extend the current base set of available APIs with two more: one for integrating the same kind of smart replies that you’re probably familiar with from apps like Inbox and Gmail, and a high-density face contour feature for the face detection API.

The real game-changer here are the offline models that developers can integrate into their apps and that they can use for free. Unsurprisingly, there is a trade-off here. The models that run on the device are smaller and hence offer a lower level of accuracy. In the cloud, neither model size nor available compute power are an issue, so those models are larger and hence more accurate, too.

That’s pretty much standard for the industry. Earlier this year, Microsoft launched its offline neural translations, for example, which can also either run online or on the device. The trade-off there is the same.

Brahim Elbouchikhi, Google’s group product manager for machine intelligence and the camera lead for Android, told me that a lot of developers will likely do some of the preliminary machine learning inference on the device, maybe to see if there is actually an animal in a picture, and then move to the cloud to detect what breed of dog it actually is. And that makes sense, because the on-device image labeling service features about 400 labels, while the cloud-based one features more than 10,000. To power the on-device models, ML Kit uses the standard Neural Networks API on Android and its equivalent on Apple’s iOS.

He also stressed that this is very much a cross-platform product. Developers don’t think of machine learning models as Android- or iOS-specific, after all.

For developers who want to go beyond the pre-trained models, ML Kit also supports TensorFlow Lite models.

As Google rightly notes, getting up to speed with using machine learning isn’t for the faint of heart. This new SDK, which falls under Google’s Firebase brand, is clearly meant to make using machine learning easier for mobile developers. While Google Cloud already offers a number of similar pre-trained and customizable machine learning APIs, those don’t work offline, and the experience isn’t integrated tightly with Firebase and the Firebase Console either, which is quickly becoming Google’s go-to hub for all things mobile development.

Even for custom TensorFlow Lite models, Google is working on compressing the models to a more workable size. For now, this is only an experiment, but developers who want to give it a try can sign up here.

Overall, Elbouchikhi argues, the work here is about democratizing machine learning. “Our goal is to make machine learning just another tool,” he said.

Android Text Recognition from Camera - Android Studio Tutorial Text Recogniser using ML KIT

Realtime Selfie Segmentation In Android With MLKit

In order to add the MLKit Selfie Segmentation feature to our Android app, we need to add a dependency to our file.

Note, make sure to use the latest release of the Selfie Segmentation package.

As we’ll perform image segmentation on the live camera feed, we’ll also require a camera library. So, in order to use CameraX, we add the following dependencies in the same file.

Note, make sure to use the latest release of the CameraX package.

Build and sync the project to make sure we’re to go!

In order to display the live camera feed to the user, we’ll use from the CameraX package. We’ll require minimum setup to get a camera live feed running because of .

Now, head on to and delete the which is present there ( the default showing ‘Hello World’ ). Next, add in .

We need to initialize the in , but first we need to add the permission to like below,

Now, open up and in the method, check if the camera permission is granted, if not, request it from the user. To provide a full screen experience to the user, remove the status bar as well. Also, initialize the we created in .

To request the camera permission, we’ll use so that the request code is automatically handled by the system. If the permission is denied, we’ll display an to user,

Wondering what the method will would do? It simply starts the live camera feed using the , which we initialized earlier and a object,

Now, run the app on a device/emulator and grant the camera permission to the app. The camera feed should run as expected. This completes half of our journey, as we still have to display a segmentation map to the user.

In order to display the segmentation over the live camera feed, we’ll need a which will be placed over the in . The camera frames ( as ) will be supplied to the overlay so that they could be drawn over the live camera feed. To start, we create a custom called which inherits .

We’ll add the above element in .

Also, we need to initialize the in the which we will help us connect it with the live camera feed:

In order to perform segmentation and display the output to the user, we first need a way to get the camera frames from the live feed. Going through the CameraX documentation, you’ll notice that we have to use in order to get the camera frames as which can be converted to our favorite .

We then create a new class, which inherits and takes the as an argument in its constructor. We’ll discuss this in the next section, as this will help us connect the with the live camera feed.

We’ll finally initialize which will segment the images from us. For every image-based service in MLKit, you need to convert the input image ( which can be a , or ) to which comes from the MLKit package. All the above mentioned logic will be executed in method. We’ll use the method to directly use the object provided by the method.

In the above code snippet, we convert which is a to . Finally, we assign the value of to the variable present in . We will also call to refresh the overlay. This calls the in the class, were we will display the segmentation to the user in a later section.

This connects the live camera feed to the with the help of . One last thing, we need to attach with Camera in ,

As we saw in the implementation of the class, there’s a variable which holds the segmentation bitmap for the current frame. Our goal is to draw this onto the screen. So, we call in the method of our ,

Also, note that we need to use method as we’ll obtain a mirror image of the segmentation.

That’s all, we’re done! Run the app on a physical device and see the magic happen right in front of your eyes!

Hope you loved MLKit’s Segmentation API. For any suggestions & queries, feel free to write a message on [email protected] ( including the story’s link/title ). Keep reading, keep learning and have a nice day ahead!


Android mlkit

A Look at Android ML Kit - the Machine Learning SDK

TL;DR: In today's fast-moving, information-rich world, it is becoming more necessary to build applications that are intelligent in the way they process the data they are fed. Artificial Intelligence is quickly becoming an essential tool in software development. In this article, we will look at the ML Kit mobile SDK that brings all the Google’s expertise on machine learning techniques to mobile developers in an easy-to-use package. We will look at the various APIs offered by the SDK, and then we will take one of the APIs on a test drive by creating an Android application that makes use of it. You can find the code for the application in this GitHub repository.

"Google I/O 2018 enabled @Android developers to take advantage of some cool #MachineLearning APIs. Learn what are the new APIs and create a simple app that recognizes objects on images."


Tweet This


In today's information-rich world, people have come to expect their technology to be smart. We are seeing the increased adoption of Artificial Intelligence(AI) in the development of intelligent software. AI is quickly becoming an essential tool in software development.

Luckily for developers, there are various services that make it easier and faster to add Artificial Intelligence to apps without needing much experience in the field. There has been a growing number of AI-related APIs in the market such as Amazon's AWS Machine Learning APIs, IBM Watson and Google Cloud AI. In this article, we'll take a look at the ML Kit mobile SDK that was introduced at this year's Google IO.

ML Kit is a mobile SDK that enables you to add powerful machine learning features to a mobile application. It supports both Android and iOS and offers the same features for both platforms. The SDK is part of Firebase and bundles together various machine learning technologies from Google such as the Cloud Vision API, Android Neural Network API and TensorFlow Lite. This SDK comes with a set of ready-to-use APIs for common mobile use cases such as face detection, text recognition, barcode scanning, image labeling and landmark recognition. These are offered as either on-device or cloud APIs. On-device APIs have the advantage of being able to process data quickly, they are free to use and they don't require a network connection to work. The cloud-based APIs give a higher level of accuracy as they are able to leverage the power of Google Cloud Platform's machine learning technologies. All cloud-based APIs are premium services, with a free quota in place.

In this article, we'll briefly go over what each of the ML Kit APIs offers before taking a look at how to use one of the APIs in an Android application.

Text Recognition with ML Kit SDK

With the text recognition API, your app can recognize text in any Latin-based language (and more when using the Cloud-based API). This can have such use cases as automating data entry from physical records to digital format, providing better accessibility where apps can identify text in images and read it out to users, organize photos based on their text content, e.t.c.

Text recognition is available both as an on-device and cloud-based API. The on-device API provides real-time processing (ideal for a camera or video feed) while the cloud-based one provides higher accuracy text recognition and is able to identify a broader range of languages and special characters.

Face Detection with ML Kit SDK

The face detection API can detect human faces in visual media (digital images and video). Given an image, the API returns the position, size and orientation (the angle the face is oriented with respect to the camera) of any detected faces. For each detected face, you can also get landmark and classification information. Landmarks are points of interest within a face such as right eye, left eye, nose base, bottom mouth, e.t.c. Classification determines whether the face displays certain facial characteristics. ML Kit currently supports two classifications: eyes open and smiling. The API is available on-device.

"@Android developers can now detect faces with ease using the new #MachineLearning SDK introduced by Google."


Tweet This

Barcode Scanning with ML Kit SDK

With the barcode scanning API, your app can read data encoded using most standard barcode formats. It is available on-device and supports the following barcode formats:

  • 1D barcodes: EAN-13, EAN-8, UPC-A, UPC-E, Code-39, Code-93, Code-128, ITF, Codabar
  • 2D barcodes: QR Code, Data Matrix, PDF-417, AZTEC

    The Barcode Scanning API automatically parses structured data stored using one of the supported 2D formats. Supported information types include:

  • URLs
  • Contact information (VCARD, etc.)
  • Calendar events
  • Email addresses
  • Phone numbers
  • SMS message prompts
  • ISBNs
  • WiFi connection information
  • Geo-location (latitude and longitude)
  • AAMVA-standard driver information (license/ID)

    Image Labeling with ML Kit SDK

The image labeling API can recognize entities in an image. When used, the API returns a list of recognized entities, each with a score indicating the confidence the ML model has in its relevance. The API can be used for such tasks as automatic metadata generation and content moderation.

Image labeling is available both as an on-device and cloud-based API. The device-based API supports 400+ labels that cover the most commonly found concepts in photos (see examples) while the cloud-based API supports 10,000+ labels (see examples).

Landmark Recognition with ML Kit SDK

The landmark recognition API can recognize well-known landmarks in an image. When given an image, the API returns landmarks that were recognized, coordinates of the position of each landmark in the image and each landmark's geographic coordinates. The API can be used to generate metadata for images or to customize some features according to the content a user shares. Landmark recognition is only available as a cloud-based API.

Using Custom Models with ML Kit SDK

If you are an experienced machine learning engineer and would prefer not to use the pre-built ML Kit models, you can use your own custom TensorFlow Lite models with ML Kit. The models can either be hosted on Firebase or they can be bundled with the app. Hosting the model on Firebase reduces your app's binary size while also ensuring that the app is always working with the most up-to-date version of the model. Storing the model locally on the device makes for faster processing. You can choose to support both on-device and cloud-hosted models in your app. By using both, you make the most recent version of the model available to your app while also ensuring that the app's ML features are always functional even if the Firebase-hosted model is unavailable (perhaps due to network issues).

Upcoming APIs

When ML Kit was released, Google also announced its plans of releasing two more APIs in the near future. These are the Smart Reply and Face Contour APIs.

The Smart Reply API will allow you to support contextual messaging replies in your app. The API will provide suggested text snippets that fit the context of messages it is sent, similar to the suggested-response feature we see in the Android Messages app.

The Face Contour API will be an addition to the Face Detection API. It will provide a high-density face contour. This will enable you to perform much more precise operations on faces than you can with the Face Detection API. To see a preview of the API in use, you can take a look at this YouTube video.

Summary of On-Device and In-Cloud Features

Google's mobile machine learning kit features

Image Labeling in an Android App

To see one of the APIs in action, we will create an application that uses the Image Labeling API to identify the contents of an image. The APIs share some similarities when it comes to integration, so knowing how to use one can help you understand how to implement the others.

To get started, create a new project in Android Studio. Give your application a name; I named mine . Firebase features are only available on devices running API level 14 and above, so select 14 or higher for your app's minimum SDK. On the next window, select the template and on the last one, you can leave the default Activity name of .

Firebase Basic Activity Template

To add Firebase to your app, first, create a Firebase project in the Firebase console.

Create Firebase Project

On the dashboard, select .

Add Firebase to your Android app

Fill out the provided form with your app's details. For this project, you only need to provide a package name (you can find this in your Android project's file). You can add an app nickname to easily identify the application on the Firebase console. This can come in handy if you are going to add several applications to the same project.

Firebase app registration form

After the app has been registered, you will see a button you can use to download a config file named . Download the file and move it into your Android app module root directory. This is where the file should be located if you use the Project view in Android Studio.

Android Studio Project Directory

The Google services plugin for Gradle loads the file that you added to your project. Modify your Project level file to use the plugin by adding it to .

Next, add the following dependencies to the App-level file (the one found in the directory).

Then add the following to the bottom of the same file (right after dependencies) and press 'Sync now' in the bar that appears in the IDE.

Open up the file and add the following string values that we'll use in the app.

In add the following item to the menu. We'll see what this is for, later on.

The app will allow the user to select an image on their phone and process it with the ML Kit library. To load images, it will require permission to read the phone's storage. Add the following permission to your file.

Also, add the following to the manifest file inside

The above code is optional but recommended to add to your manifest file if your app will use any of the on-device APIs. With the above configuration, the app will automatically download the ML model(s) specified by to the device soon after the app is installed from the Play Store. If you don't enable install-time model downloads, then the model that app needs will be downloaded the first time the app runs the detector for that particular model. Requests made before the download is complete won't produce any results. To use multiple models, include them in a comma-separated list, e.g. .

Replace the content of with the following code.

In , give the root an :

Then change the icon of the FloatingActionButton from to .

Add the following variables to the class.

If you haven't done so, I recommend enabling Auto Import on Android Studio which will automatically import unambiguous libraries to the class as you add code that uses them. You can also refer to this file for a full list of libraries used in .

Modify as shown below.

Here, we instantiate view objects in our layout and set a click listener on the FloatingActionButton. The user will use this button to select an image from their phone. The image will then be loaded onto the ImageView that we added to .

Add the following two functions to the class.

In we check if the user has granted the app permission to read from storage. If yes, we call which opens up an Activity that the user can use to select a photo. returns if the user previously denied the permission request and returns if they had denied the permission and checked the Don't ask again option in the request dialog. If they had previously denied permission but had not checked the Don't ask again option, then we show them a Snackbar with an explanation of why the app needs the permission. On clicking on OK on that Snackbar, the permission request dialog will be shown to the user.

If the user had denied the permission and chosen the Don't ask again option, the Android system respects their request and doesn't show them the permission dialog again for that specific permission. In our code, we check for this and call . This method usually brings up the permission request dialog, but since the user requested not to be asked again for that permission, the system itself denies the permission without showing the dialog. The system calls callback method and passes , the same way it would if the user had explicitly rejected the request.

Add the following methods to .

In , we called which starts an Activity that enables the user to select an image. When the user is done with that activity, the system calls . Here, we grab the image that the user selected and set it onto the ImageView in our layout. We also set the value of the TextView to an empty string. Later on, we will display the result of ML Kit's image labeling on this TextView. So for each image selected, we clear the TextView in case it was displaying the results of a previous image.

is the callback that is invoked when permissions are requested. Here we check for the result of the permission that was requested with the code and then check if the permission was granted or not. If the permission was granted, then we call which we have looked at previously. If the permission was denied, either by the user or by the system as a result of the user having previously chosen not to be asked again, it is recommended that you gracefully handle this in a way that will not cause the program to crash or to not work correctly. You can, for instance, disable the functionality that requires that permission and then let the user know why it is disabled. If the user requested to not be asked again for the permission, you can include in your message instructions for them on how to enable permission via Settings.

After the user selects an image, we want them to be able to process it with ML Kit. In we had added a menu item to the layout. Add an clause to the statement in the method in with the following code that calls when the user taps the menu item.

Next, add the following to the class.

When is called, we first check if the ImageView has an image. If it doesn't, we let the user know that an image is required, otherwise, we process the image with ML Kit.

Before we can label the image, we first create a object from the image. The object can be created from either a , , , byte array, or a file on the device.

We then instantiate the object that we had declared earlier. is a detector for finding s in a supplied image.

Finally, we pass the image to the method and register listeners that will be called when the detection completes.

If the labeling succeeds, a list of objects is passed to the success listener. Each object represents a label for the image. For each label, you can get its text description, its Knowledge Graph entity ID (if available) and its confidence score. We get these, append them to a string and finally display the string in the TextView we added to the app.

When you are done with the detector or when it is clear that it will not be in use, you should close it to release the resources it is using. In our app, we can do this in which will close the detector when the activity goes to the background.

With that, you can now run the application.

On selecting an image and tapping on the Process button on the App Bar, you will be able to see the various labels and their corresponding confidence scores. If you are running the app on the emulator refer to this guide to add an image to the app's gallery.

ML Kit Image Processing Results

By default, the on-device image labeler returns a maximum of 10 labels (returned labels can be less than 10). If you want to change this setting, you can set a confidence threshold on your detector. This is done with the object. As an example, you can specify that the detector only returns labels that are above a certain level. Below, we specify that the detector only returns labels with a confidence score of and above.

The object can then be passed to the detector during instantiation.

"I just built an @Android app that recognizes objects on images! #MachineLearning"


Tweet This

Aside: Securing Android Apps with Auth0

Securing applications with Auth0 is very easy and brings a lot of great features to the table. With Auth0, we only have to write a few lines of code to get solid identity management solution, single sign-on, support for social identity providers (like Facebook, GitHub, Twitter, etc.), and support for enterprise identity providers (Active Directory, LDAP, SAML, custom, etc.).

In the following sections, we are going to learn how to use Auth0 to secure Android apps. As we will see, the process is simple and fast.


To secure Android apps with Auth0, we just need to import the Auth0.Android library. This library is a toolkit that let us communicate with many of the basic Auth0 API functions in a neat way.

To import this library, we have to include the following dependency in our file:

After that, we need to open our app's file and add the following permission:

Create an Auth0 Application

After importing the library and adding the permission, we need to register the application in our Auth0 dashboard. By the way, if we don't have an Auth0 account, this is a great time to create a free one .

In the Auth0 dashboard, we have to go to Applications and then click on the Create Application button. In the form that is shown, we have to define a name for the application and select the Native type for it. After that, we can hit the Create button. This will lead us to a screen similar to the following one:

Android application on Auth0's dashboard

On this screen, we have to configure a callback URL. This is a URL in our Android app where Auth0 redirects the user after they have authenticated.

We need to whitelist the callback URL for our Android app in the Allowed Callback URLs field in the Settings page of our Auth0 application. If we do not set any callback URL, our users will see a mismatch error when they log in.

Let's not forget to replace OURAPPPACKAGE_NAME with our Android application's package name. We can find this name in the attribute of the file.

Set Credentials

Our Android application needs some details from Auth0 to communicate with it. We can get these details from the Settings section for our Auth0 application in the Auth0 dashboard.

We need the following information:

It's suggested that we do not hardcode these values as we may need to change them in the future. Instead, let's use String Resources, such as , to define the values.

Let's edit our file as follows:

These values have to be replaced by those found in the Settings section of our Auth0 application.

Android Login

To implement the login functionality in our Android app, we need to add manifest placeholders required by the SDK. These placeholders are used internally to define an that captures the authentication callback URL configured previously.

To add the manifest placeholders, let's add the next line:

After that, we have to run Sync Project with Gradle Files inside Android Studio or execute from the command line.

Start the Authentication Process

The Auth0 login page is the easiest way to set up authentication in our application. It's recommended using the Auth0 login page for the best experience, best security, and the fullest array of features.

Now we have to implement a method to start the authentication process. Let's call this method and add it to our class.

As we can see, we had to create a new instance of the Auth0 class to hold user credentials. We can use a constructor that receives an Android Context if we have added the following String resources:

    If we prefer to hardcode the resources, we can use the constructor that receives both strings. Then, we can use the class to authenticate with any connection enabled on our application in the Auth0 dashboard.

    After we call the function, the browser launches and shows the Auth0 login page. Once the user authenticates, the callback URL is called. The callback URL contains the final result of the authentication process.

    Capture the Result

    After authentication, the browser redirects the user to our application with the authentication result. The SDK captures the result and parses it.

    We do not need to declare a specific for our activity because we have defined the manifest placeholders with we Auth0 Domain and Scheme values.

    The file should look like this:

    That's it, we now have an Android application secured with Auth0. To learn more about this, we can check the official documentation. There, we will find more topics like Session Handling and Fetching User Profile.


    In this article, we have looked at the Google ML Kit Mobile (the machine learning SDK) and all the features it offers. We've also looked at how to use one of its APIs in an Android application. To find out more about this SDK, you can watch this video from Google IO and you should also take a look at the documentation which covers all the APIs and shows their use on both Android and iOS. You can also take a look at the code in these apps, to see how the various APIs are implemented on Android and on iOS.

    • Twitter icon
    • LinkedIn icon
    • Faceboook icon
    ML Kit: Machine Learning for Mobile with Firebase (Google I/O'19)

    ML Kit for iOS and Android Now Generally Available

    After two years in beta, Google has announced the general availability of ML Kit for iOS and Android along with improvements to the Pose Detection API. Furthermore, Selfie Segmentation is now available in public beta.

    We launched ML Kit back in 2018 in order to make it easy for developers on Android and iOS to use machine learning within their apps. Over the last two years we have rapidly expanded our set of APIs to help with both vision and natural language processing based use cases.

    ML Kit vision-related stable APIs cover text recognition, face detection, barcode scanning, image labelling, oject detection and tracking, and digital ink recognition. Natural-language capabilities include language identification, text translation, and smart replies.

    Among the latest features added to ML Kit are Selfie Segmentation, Pose Detection, and Entity Extraction, which are still considered beta-quality, though.

    Selfie Segmentation, which was previously available only in closed beta, enables separating the background of a selfie from the people in the foreground, which can be useful to apply visual effects or replace the background altogether. Selfie segmentation can be applied to both still photos and video footage.

    The Selfie Segmentation API takes an input image and produces an output mask. Each pixel of the mask is assigned a float number that has a range between [0.0, 1.0]. The closer the number is to 1.0, the higher the confidence that the pixel represents a person, and vice versa.

    Since its initial announcement, ML Kit Pose Detection API, which is able to provide a 33-point skeletal match of a user's body, has been improved by making it capable of recognizing more poses, including typical fitness and yoga poses. Additionally, the new API uses smaller models, which are half the original size, and adds Z-coordinate support to make it possible to determine whether parts of the user's body are in front or behind their hips. One noteworthy use-case using pose detection is classifying specific pose, for which Google is providing a complete tutorial and sample app that also shows how to count repetitions using the classifier.

    To get started with ML Kit, head to the samples section of Google's ML Kit developer portal.

    This content is in the Mobile topic

    Related Topics:

    You will also like:


    659 660 661 662 663