Android - Face Detection

Question

Is there a way to obtain the image data when using MlKitAnalyzer for face detection ? I am able to draw the bounding box and get all the contours/landmarks but now I need to do rgb calculation on the detected face, for which I can't find a way to access the imageProxy. Any help would be much appreciated. https://developer.android.com/reference/androidx/camera/mlkit/vision/MlKitAnalyzer.Result

cameraController.setImageAnalysisAnalyzer(cameraExecutor, MlKitAnalyzer(
 listOf(faceDetectionComponent.getFaceDetector()),
 COORDINATE_SYSTEM_VIEW_REFERENCED,
 cameraExecutor
 ) { result ->
 //draw the bounding box - works well!
 //need access to image data ??
 }

I need to do the below rgb calculation on the detected face but the imageProxy or any other data related to image doesn't seem to be available in MlKitAnalyzer.Result

fun Bitmap.calculateAverageColor(): RGBData {
 var r = 0
 var g = 0
 var b = 0
 val height = this.height
 val width = this.width
 var n = 0.0
 val pixels = IntArray(width * height)
 this.getPixels(pixels, 0, width, 0, 0, width, height)
 var i = 0
 while (i < pixels.size) {
 val color = pixels[i]
 r += Color.red(color)
 g += Color.green(color)
 b += Color.blue(color)
 n++
 i += PIXEL_SPACING
 }
 return RGBData(r.toDouble() / n, g.toDouble() / n, b.toDouble() / n)
}

• VishV: Kindly provide some information about what kind of image info you want to process. Mention your full code for better understanding.
• Nimmi Mathew: @VishV I have edited this question with more information and code.
• VishV: Try my answer and check it's working or not.

Answer 1

Try this solution,

Configure MlKitAnalyzer so you get access to ImageProxy in the result callback and convert ImageProxy to Bitmap and Crop to Face Region.

cameraController.setImageAnalysisAnalyzer(cameraExecutor, MlKitAnalyzer(
 listOf(faceDetectionComponent.getFaceDetector()),
 COORDINATE_SYSTEM_VIEW_REFERENCED,
 cameraExecutor
) { result ->
 // Get the detected faces (assuming a single face for simplicity)
 val faceList = result.getValue(faceDetectionComponent.getFaceDetector())
 if (faceList != null && faceList.isNotEmpty()) {
 val face = faceList[0] // Get the first detected face
 val boundingBox = face.boundingBox
 // Access the ImageProxy
 val imageProxy = result.imageProxy
 if (imageProxy != null) {
 // Convert ImageProxy to Bitmap
 val bitmap = imageProxy.toBitmap()
 // Crop the Bitmap to the bounding box of the face
 val faceBitmap = Bitmap.createBitmap(
 bitmap,
 boundingBox.left,
 boundingBox.top,
 boundingBox.width(),
 boundingBox.height()
 )
 // Calculate the average color on the cropped face bitmap
 val avgColor = faceBitmap.calculateAverageColor()
 
 // Use avgColor as needed (e.g., log, display, etc.)
 // Don't forget to close the imageProxy
 imageProxy.close()
 }
 }
})

To convert ImageProxy to Bitmap, here’s a utility function:

fun ImageProxy.toBitmap(): Bitmap {
 val yBuffer = planes[0].buffer
 val uBuffer = planes[1].buffer
 val vBuffer = planes[2].buffer
 val ySize = yBuffer.remaining()
 val uSize = uBuffer.remaining()
 val vSize = vBuffer.remaining()
 val nv21 = ByteArray(ySize + uSize + vSize)
 yBuffer.get(nv21, 0, ySize)
 vBuffer.get(nv21, ySize, vSize)
 uBuffer.get(nv21, ySize + vSize, uSize)
 val yuvImage = YuvImage(nv21, ImageFormat.NV21, width, height, null)
 val out = ByteArrayOutputStream()
 yuvImage.compressToJpeg(Rect(0, 0, width, height), 100, out)
 val imageBytes = out.toByteArray()
 return BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)
}

• Nimmi Mathew: Sorry, it's not working. When prompted Gemini - that too gave me a similar response. But result.imageProxy doesn't exist!