Subject segmentation with ML Kit for Android

  • Easily add subject segmentation to your Android app using the ML Kit Subject Segmentation API.

  • This API requires Android API level 24 or higher and uses an unbundled library downloaded via Google Play services.

  • The model can be automatically downloaded after app installation or explicitly requested using the ModuleInstallClient API.

  • Process images by creating an InputImage object and passing it to the SubjectSegmenter, enabling options like foreground masks and bitmaps.

  • Retrieve segmentation results using methods such as getForegroundConfidenceMask(), getForegroundBitmap(), and access subject-specific masks and bitmaps.

Use the ML Kit to easily add subject segmentation features to your app.

Feature Details
Sdk name play-services-mlkit-subject-segmentation
Implementation Unbundled: the model is dynamically downloaded using Google Play services.
App size impact ~200 KB size increase.
Initialization time Users might have to wait for the model to download before first use.

Try it out

  • Play around with the sample app to see an example usage of this API.

Before you begin

  1. In your project-level build.gradle file, make sure to include Google's Maven repository in both your buildscript and allprojects sections.
  2. Add the dependency for the ML Kit subject segmentation library to your module's app-level gradle file, which is usually app/build.gradle:
dependencies{
implementation'com.google.android.gms:play-services-mlkit-subject-segmentation:16.0.0-beta1'
}

As mentioned above the model is provided by Google Play services. You can configure your app to automatically download the model to the device after your app is installed from the Play Store. To do so, add the following declaration to your app's AndroidManifest.xml file:

<application ...>
 ...
 <meta-data
 android:name="com.google.mlkit.vision.DEPENDENCIES"
 android:value="subject_segment" >
 <!-- To use multiple models: android:value="subject_segment,model2,model3" -->
</application>

You can also explicitly check the model availability and request download through Google Play services with ModuleInstallClient API.

If you don't enable install-time model downloads or request explicit download the model is downloaded the first time you run the segmenter. Requests you make before the download has completed produce no results.

1. Prepare the input image

To perform segmentation on an image, create an InputImage object from either a Bitmap, media.Image, ByteBuffer, byte array, or a file on the device.

You can create an InputImage object from different sources, each is explained below.

Using a media.Image

To create an InputImage object from a media.Image object, such as when you capture an image from a device's camera, pass the media.Image object and the image's rotation to InputImage.fromMediaImage().

If you use the CameraX library, the OnImageCapturedListener and ImageAnalysis.Analyzer classes calculate the rotation value for you.

Kotlin

privateclassYourImageAnalyzer:ImageAnalysis.Analyzer{
overridefunanalyze(imageProxy:ImageProxy){
valmediaImage=imageProxy.image
if(mediaImage!=null){
valimage=InputImage.fromMediaImage(mediaImage,imageProxy.imageInfo.rotationDegrees)
// Pass image to an ML Kit Vision API
// ...
}
}
}

Java

privateclass YourAnalyzerimplementsImageAnalysis.Analyzer{
@Override
publicvoidanalyze(ImageProxyimageProxy){
ImagemediaImage=imageProxy.getImage();
if(mediaImage!=null){
InputImageimage=
InputImage.fromMediaImage(mediaImage,imageProxy.getImageInfo().getRotationDegrees());
// Pass image to an ML Kit Vision API
// ...
}
}
}

If you don't use a camera library that gives you the image's rotation degree, you can calculate it from the device's rotation degree and the orientation of camera sensor in the device:

Kotlin

privatevalORIENTATIONS=SparseIntArray()
init{
ORIENTATIONS.append(Surface.ROTATION_0,0)
ORIENTATIONS.append(Surface.ROTATION_90,90)
ORIENTATIONS.append(Surface.ROTATION_180,180)
ORIENTATIONS.append(Surface.ROTATION_270,270)
}
/**
 * Get the angle by which an image must be rotated given the device's current
 * orientation.
 */
@RequiresApi(api=Build.VERSION_CODES.LOLLIPOP)
@Throws(CameraAccessException::class)
privatefungetRotationCompensation(cameraId:String,activity:Activity,isFrontFacing:Boolean):Int{
// Get the device's current rotation relative to its "native" orientation.
// Then, from the ORIENTATIONS table, look up the angle the image must be
// rotated to compensate for the device's rotation.
valdeviceRotation=activity.windowManager.defaultDisplay.rotation
varrotationCompensation=ORIENTATIONS.get(deviceRotation)
// Get the device's sensor orientation.
valcameraManager=activity.getSystemService(CAMERA_SERVICE)asCameraManager
valsensorOrientation=cameraManager
.getCameraCharacteristics(cameraId)
.get(CameraCharacteristics.SENSOR_ORIENTATION)!!
if(isFrontFacing){
rotationCompensation=(sensorOrientation+rotationCompensation)%360
}else{// back-facing
rotationCompensation=(sensorOrientation-rotationCompensation+360)%360
}
returnrotationCompensation
}

Java

privatestaticfinalSparseIntArrayORIENTATIONS=newSparseIntArray();
static{
ORIENTATIONS.append(Surface.ROTATION_0,0);
ORIENTATIONS.append(Surface.ROTATION_90,90);
ORIENTATIONS.append(Surface.ROTATION_180,180);
ORIENTATIONS.append(Surface.ROTATION_270,270);
}
/**
 * Get the angle by which an image must be rotated given the device's current
 * orientation.
 */
@RequiresApi(api=Build.VERSION_CODES.LOLLIPOP)
privateintgetRotationCompensation(StringcameraId,Activityactivity,booleanisFrontFacing)
throwsCameraAccessException{
// Get the device's current rotation relative to its "native" orientation.
// Then, from the ORIENTATIONS table, look up the angle the image must be
// rotated to compensate for the device's rotation.
intdeviceRotation=activity.getWindowManager().getDefaultDisplay().getRotation();
introtationCompensation=ORIENTATIONS.get(deviceRotation);
// Get the device's sensor orientation.
CameraManagercameraManager=(CameraManager)activity.getSystemService(CAMERA_SERVICE);
intsensorOrientation=cameraManager
.getCameraCharacteristics(cameraId)
.get(CameraCharacteristics.SENSOR_ORIENTATION);
if(isFrontFacing){
rotationCompensation=(sensorOrientation+rotationCompensation)%360;
}else{// back-facing
rotationCompensation=(sensorOrientation-rotationCompensation+360)%360;
}
returnrotationCompensation;
}

Then, pass the media.Image object and the rotation degree value to InputImage.fromMediaImage():

Kotlin

valimage=InputImage.fromMediaImage(mediaImage,rotation)

Java

InputImageimage=InputImage.fromMediaImage(mediaImage,rotation);

Using a file URI

To create an InputImage object from a file URI, pass the app context and file URI to InputImage.fromFilePath(). This is useful when you use an ACTION_GET_CONTENT intent to prompt the user to select an image from their gallery app.

Kotlin

valimage:InputImage
try{
image=InputImage.fromFilePath(context,uri)
}catch(e:IOException){
e.printStackTrace()
}

Java

InputImageimage;
try{
image=InputImage.fromFilePath(context,uri);
}catch(IOExceptione){
e.printStackTrace();
}

Using a ByteBuffer or ByteArray

To create an InputImage object from a ByteBuffer or a ByteArray, first calculate the image rotation degree as previously described for media.Image input. Then, create the InputImage object with the buffer or array, together with image's height, width, color encoding format, and rotation degree:

Kotlin

valimage=InputImage.fromByteBuffer(
byteBuffer,
/* image width */480,
/* image height */360,
rotationDegrees,
InputImage.IMAGE_FORMAT_NV21// or IMAGE_FORMAT_YV12
)
// Or:
valimage=InputImage.fromByteArray(
byteArray,
/* image width */480,
/* image height */360,
rotationDegrees,
InputImage.IMAGE_FORMAT_NV21// or IMAGE_FORMAT_YV12
)

Java

InputImageimage=InputImage.fromByteBuffer(byteBuffer,
/* image width */480,
/* image height */360,
rotationDegrees,
InputImage.IMAGE_FORMAT_NV21// or IMAGE_FORMAT_YV12
);
// Or:
InputImageimage=InputImage.fromByteArray(
byteArray,
/* image width */480,
/* image height */360,
rotation,
InputImage.IMAGE_FORMAT_NV21// or IMAGE_FORMAT_YV12
);

Using a Bitmap

To create an InputImage object from a Bitmap object, make the following declaration:

Kotlin

valimage=InputImage.fromBitmap(bitmap,0)

Java

InputImageimage=InputImage.fromBitmap(bitmap,rotationDegree);

The image is represented by a Bitmap object together with rotation degrees.

2. Create an instance of SubjectSegmenter

Define the segmenter options

To segment your image, first create an instance of SubjectSegmenterOptions as follow:

Kotlin

val options = SubjectSegmenterOptions.Builder()
 // enable options
 .build()

Java

SubjectSegmenterOptions options = new SubjectSegmenterOptions.Builder()
 // enable options
 .build();

Here is the detail of each option:

Foreground confidence mask

The foreground confidence mask lets you distinguish the foreground subject from the background.

Call enableForegroundConfidenceMask() in the options lets you later retrieve the foreground mask by calling getForegroundMask() on the SubjectSegmentationResult object returned after processing the image.

Kotlin

valoptions=SubjectSegmenterOptions.Builder()
.enableForegroundConfidenceMask()
.build()

Java

SubjectSegmenterOptionsoptions=newSubjectSegmenterOptions.Builder()
.enableForegroundConfidenceMask()
.build();
Foreground bitmap

Similarly, you can also get a bitmap of the foreground subject.

Call enableForegroundBitmap() in the options lets you to later retrieve the foreground bitmap by calling getForegroundBitmap() on the SubjectSegmentationResult object returned after processing the image.

Kotlin

valoptions=SubjectSegmenterOptions.Builder()
.enableForegroundBitmap()
.build()

Java

SubjectSegmenterOptionsoptions=newSubjectSegmenterOptions.Builder()
.enableForegroundBitmap()
.build();
Multi-subject confidence mask

Like for the foreground options, you can use the SubjectResultOptions to enable the confidence mask for each foreground subject as follow:

Kotlin

val subjectResultOptions = SubjectSegmenterOptions.SubjectResultOptions.Builder()
 .enableConfidenceMask()
 .build()
val options = SubjectSegmenterOptions.Builder()
 .enableMultipleSubjects(subjectResultOptions)
 .build()

Java

SubjectResultOptions subjectResultOptions =
 new SubjectSegmenterOptions.SubjectResultOptions.Builder()
 .enableConfidenceMask()
 .build()
SubjectSegmenterOptions options = new SubjectSegmenterOptions.Builder()
 .enableMultipleSubjects(subjectResultOptions)
 .build()
Multi-subject bitmap

And similarly, you can enable the bitmap for each subject:

Kotlin

val subjectResultOptions = SubjectSegmenterOptions.SubjectResultOptions.Builder()
 .enableSubjectBitmap()
 .build()
val options = SubjectSegmenterOptions.Builder()
 .enableMultipleSubjects(subjectResultOptions)
 .build()

Java

SubjectResultOptions subjectResultOptions =
 new SubjectSegmenterOptions.SubjectResultOptions.Builder()
 .enableSubjectBitmap()
 .build()
SubjectSegmenterOptions options = new SubjectSegmenterOptions.Builder()
 .enableMultipleSubjects(subjectResultOptions)
 .build()

Create the subject segmenter

Once you specified the SubjectSegmenterOptions options, create a SubjectSegmenter instance calling getClient() and passing the options as a parameter:

Kotlin

val segmenter = SubjectSegmentation.getClient(options)

Java

SubjectSegmenter segmenter = SubjectSegmentation.getClient(options);

3. Process an image

Pass the prepared InputImage object to the SubjectSegmenter's process method:

Kotlin

segmenter.process(inputImage)
.addOnSuccessListener{result->
// Task completed successfully
// ...
}
.addOnFailureListener{e->
// Task failed with an exception
// ...
}

Java

segmenter.process(inputImage)
.addOnSuccessListener(newOnSuccessListener(){
@Override
publicvoidonSuccess(SubjectSegmentationResultresult){
//Taskcompletedsuccessfully
//...
}
})
.addOnFailureListener(newOnFailureListener(){
@Override
publicvoidonFailure(@NonNullExceptione){
//Taskfailedwithanexception
//...
}
});

4. Get the subject segmentation result

Retrieve foreground masks and bitmaps

Once processed, you can retrieve the foreground mask for your image calling getForegroundConfidenceMask() as follow:

Kotlin

valcolors=IntArray(image.width*image.height)
valforegroundMask=result.foregroundConfidenceMask
for(iin0untilimage.width*image.height){
if(foregroundMask[i]>0.5f){
colors[i]=Color.argb(128,255,0,255)
}
}
valbitmapMask=Bitmap.createBitmap(
colors,image.width,image.height,Bitmap.Config.ARGB_8888
)

Java

int[]colors=newint[image.getWidth() * image.getHeight()];
FloatBufferforegroundMask=result.getForegroundConfidenceMask();
for(inti=0;i<image.getWidth()*image.getHeight();i++){
if(foregroundMask.get()>0.5f){
colors[i]=Color.argb(128,255,0,255);
}
}
BitmapbitmapMask=Bitmap.createBitmap(
colors,image.getWidth(),image.getHeight(),Bitmap.Config.ARGB_8888
);

You can also retrieve a bitmap of the foreground of the image calling getForegroundBitmap():

Kotlin

valforegroundBitmap=result.foregroundBitmap

Java

BitmapforegroundBitmap=result.getForegroundBitmap();

Retrieve masks and bitmaps for each subject

Similarly, you can retrieve the mask for the segmented subjects by calling getConfidenceMask() on each subject as follow:

Kotlin

valsubjects=result.subjects
valcolors=IntArray(image.width*image.height)
for(subjectinsubjects){
valmask=subject.confidenceMask
for(iin0untilsubject.width*subject.height){
valconfidence=mask[i]
if(confidence>0.5f){
colors[image.width * (subject.startY - 1) + subject.startX]=
Color.argb(128,255,0,255)
}
}
}
valbitmapMask=Bitmap.createBitmap(
colors,image.width,image.height,Bitmap.Config.ARGB_8888
)

Java

List subjects = result.getSubjects();
int[] colors = new int[image.getWidth() * image.getHeight()];
for (Subject subject : subjects) {
 FloatBuffer mask = subject.getConfidenceMask();
 for (int i = 0; i < subject.getWidth() * subject.getHeight(); i++) {
 float confidence = mask.get();
 if (confidence> 0.5f) {
 colors[width * (subject.getStartY() - 1) + subject.getStartX()]
 = Color.argb(128, 255, 0, 255);
 }
 }
}
Bitmap bitmapMask = Bitmap.createBitmap(
 colors, image.width, image.height, Bitmap.Config.ARGB_8888
);

You can also access the bitmap of each segmented subject as follow:

Kotlin

val bitmaps = mutableListOf()
for (subject in subjects) {
 bitmaps.add(subject.bitmap)
}

Java

List bitmaps = new ArrayList();
for (Subject subject : subjects) {
 bitmaps.add(subject.getBitmap());
}

Tips to improve performance

For each app session, the first inference is often slower than subsequent inferences due to model initialization. If low latency is critical, consider calling a "dummy" inference ahead of time.

The quality of your results depends on the quality of the input image:

  • For ML Kit to get an accurate segmentation result, the image should be at least 512x512 pixels.
  • Poor image focus can also impact accuracy. If you don't get acceptable results, ask the user to recapture the image.

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2026年09月01日 UTC.