ZoomCallback not being invoked

Question

I am having issues with configuring ZoomSuggestionOptions for BarcodeScannerOptions. Maybe someone managed to make it work? As documentation is so badly written it's impossible to get anything from it.

Maybe it's because I use MlKitAnalyzer, which is pre-made implementation of ImageAnalysis.Analyzer, but I am not sure.

I've tried using controller.setZoomRatio() and zoom correctly is applied in LaunchedEffect block, so its not an issue with controller itself.

 val controller = remember {
 LifecycleCameraController(context).apply {
 setEnabledUseCases(
 CameraController.VIDEO_CAPTURE or
 CameraController.IMAGE_ANALYSIS or
 CameraController.IMAGE_CAPTURE
 )
 cameraSelector = CameraSelector.DEFAULT_BACK_CAMERA
 }
 }
 val zoomSuggestionOptions = remember {
 ZoomSuggestionOptions.Builder(
 object: ZoomSuggestionOptions.ZoomCallback {
 override fun setZoom(zoomRatio: Float): Boolean {
 Log.d("ZoomRatio", "setZoom: $zoomRatio")
 controller.setZoomRatio(zoomRatio)
 return true
 }
 }
 )
 .setMaxSupportedZoomRatio(controller.cameraInfo?.zoomState?.value?.maxZoomRatio ?: 0f).build()
 }
 val options = remember {
 BarcodeScannerOptions.Builder()
 .setZoomSuggestionOptions(zoomSuggestionOptions)
 .setBarcodeFormats(FORMAT_QR_CODE, FORMAT_DATA_MATRIX)
 .enableAllPotentialBarcodes()
 .build()
 }
 val scanner = remember { BarcodeScanning.getClient(options) }
 LaunchedEffect(controller) {
 controller.setImageAnalysisAnalyzer(
 ContextCompat.getMainExecutor(context.applicationContext),
 MlKitAnalyzer(
 listOf(scanner),
 ImageAnalysis.COORDINATE_SYSTEM_VIEW_REFERENCED,
 ContextCompat.getMainExecutor(context.applicationContext)
 ) { result: MlKitAnalyzer.Result? ->
 val barcodeResults = result?.getValue(scanner)
 barcodeResults?.let {
 barcodeAnalyzerOutput = barcodeAnalyzerOutput.copy(
 barcodes = it.map { barcode ->
 BarcodeDetails(
 bounds = barcode.boundingBox,
 rawValue = barcode.rawValue,
 displayValue = barcode.displayValue,
 valueType = barcode.valueType,
 corners = barcode.cornerPoints
 )
 }
 )
 }
 },
 )
 }

• user31399013: Hi, I'm encountering the same exact issue. Did you find a solution? Was it the usage of MlKitAnalyzer (which I suspect, since I had a mockup working with a custom ImageAnalyzer) and is there a fix?

Answer 1

For anyone looking for a solution:

It turns out that the MlKitAnalyzer class is the culprit. The MlKitAnalyzer accepts a generalized List<Detector<T>>, which erases the individual class types.

During analysis the analyzer therefore makes each detector process the frames by the commonly function fun Detector.process(image: Image, rotation: Int, transform: Matrix).

For some reason ZoomSuggestions don't seem to be processed, when analyzing through the above function. As the source code for the entire com.google.mlkit.vision.barcode-package (BarcodeScanner & ZoomSuggestionOptions) isn't open source, I can't tell the specific reason for this implementation.

But I can conclude that processing the ImageProxy.image with fun Detector.process(image: InputImage) invokes the ZoomSuggestionCallback. Problem solved - almost

Workarounds

The BarcodeScanner class is made private under the BarcodeScanning class, and as such can't be extended to my knowledge.

The MlKitAnalyzer class is public but the public void analyze(@NonNull ImageProxy imageProxy), is final and as such can't be overridden. Internally it calls void detectRecursively(@NonNull ImageProxy imageProxy, int detectorIndex, @NonNull Matrix transform, Map<Detector<?>, Object> values, @NonNull Map<Detector<?>, Throwable> throwables) but this function is private and can't be overridden.

Bummer!

Luckily the source code for MlKitAnalyzer is public. Maybe someone more experienced has a more elegant solution, but I could only surmise that cloning the source code and modifying it was the only "solution".

Cloned MlKitAnalyzer

Below is a cloned copy of the MlKitAnalyzer with any changed marked by: /** ADDED: <some comment> */ - I have removed most of the MlKitAnalyzer documentation, just for briefness.

/**
 * A modified copy of MlKitAnalyzer
 * */
@TransformExperimental
class CustomKitAnalyzer(
 detectorsList: List<Detector<*>>,
 private val targetCoordinateSystem: Int,
 private val executor: Executor,
 private val consumer: Consumer<Result>
) : ImageAnalysis.Analyzer {
 private var detectors: List<Detector<*>>
 val imageAnalysisTransformFactory: ImageProxyTransformFactory
 private var sensorToTarget: Matrix? = null
 init {
 if (targetCoordinateSystem != ImageAnalysis.COORDINATE_SYSTEM_ORIGINAL) {
 for (detector in detectorsList) {
 Preconditions.checkArgument(
 detector.detectorType != Detector.TYPE_SEGMENTATION,
 "Segmentation only works with COORDINATE_SYSTEM_ORIGINAL"
 )
 }
 }
 // Make an immutable copy of the app provided detectors.
 detectors = ArrayList(detectorsList)
 imageAnalysisTransformFactory = ImageProxyTransformFactory().apply {
 isUsingRotationDegrees = true
 }
 }
 @SuppressLint("RestrictedApi")
 override fun analyze(imageProxy: ImageProxy) {
 // By default, the matrix is identity for COORDINATE_SYSTEM_ORIGINAL.
 val analysisToTarget = Matrix()
 if (targetCoordinateSystem != ImageAnalysis.COORDINATE_SYSTEM_ORIGINAL) {
 // Calculate the transform if not COORDINATE_SYSTEM_ORIGINAL.
 val sensorToTarget = sensorToTarget
 if (targetCoordinateSystem != ImageAnalysis.COORDINATE_SYSTEM_SENSOR && sensorToTarget == null) {
 // If the app sets an sensor to target transformation, we cannot provide correct
 // coordinates until it is ready. Return early.
 Log.d(TAG, "Sensor-to-target transformation is null.")
 imageProxy.close()
 return
 }
 val sensorToAnalysis =
 Matrix(imageProxy.imageInfo.sensorToBufferTransformMatrix)
 // Calculate the rotation added by ML Kit.
 val sourceRect = RectF(
 0f, 0f, imageProxy.width.toFloat(),
 imageProxy.height.toFloat()
 )
 val bufferRect = TransformUtils.rotateRect(
 sourceRect,
 imageProxy.imageInfo.rotationDegrees
 )
 val analysisToMlKitRotation = TransformUtils.getRectToRect(
 sourceRect, bufferRect,
 imageProxy.imageInfo.rotationDegrees
 )
 // Concat the MLKit transformation with sensor to Analysis.
 sensorToAnalysis.postConcat(analysisToMlKitRotation)
 // Invert to get analysis to sensor.
 sensorToAnalysis.invert(analysisToTarget)
 if (targetCoordinateSystem != ImageAnalysis.COORDINATE_SYSTEM_SENSOR) {
 // Concat the sensor to target transformation to get the overall transformation.
 analysisToTarget.postConcat(sensorToTarget)
 }
 }
 // Detect the image recursively, starting from index 0.
 detectRecursively(
 imageProxy,
 0,
 analysisToTarget,
 HashMap(),
 HashMap()
 )
 }
 @OptIn(ExperimentalGetImage::class)
 private fun detectRecursively(
 imageProxy: ImageProxy,
 detectorIndex: Int,
 transform: Matrix,
 values: MutableMap<Detector<*>, Any>,
 throwables: MutableMap<Detector<*>, Throwable?>
 ) {
 val image = imageProxy.image
 if (image == null) {
 // No-op if the frame is not backed by ImageProxy.
 Log.e(TAG, "Image is null.")
 imageProxy.close()
 return
 }
 if (detectorIndex > detectors.size - 1) {
 // Termination condition is met when the index reaches the end of the list.
 imageProxy.close()
 executor.execute {
 consumer.accept(
 Result(
 values,
 imageProxy.imageInfo.timestamp,
 throwables
 )
 )
 }
 return
 }
 val detector = detectors[detectorIndex]
 val rotationDegrees = imageProxy.imageInfo.rotationDegrees
 val task: Task<*>?
 try {
 /** ADDED: check on detector type to perform special operations */
 task = when (detector.detectorType) {
 BARCODE_SCANNER_DETECTOR_TYPE -> {
 (detector as BarcodeScanner).process(InputImage.fromMediaImage(image, rotationDegrees))
 }
 else -> {
 detector.process(image, rotationDegrees, transform)
 }
 }
 } catch (e: Exception) {
 Log.w(TAG, "Detector of type ${detector.detectorType} failed to process the frame: $e")
 // If the detector is closed, it will throw a MlKitException.UNAVAILABLE. It's not
 // public in the "mlkit:vision-interfaces" artifact so we have to catch a generic
 // Exception here.
 throwables.put(detector, RuntimeException("Failed to process the image.", e))
 // This detector is closed, but the next one might still be open. Send the image to
 // the next detector.
 detectRecursively(
 imageProxy, detectorIndex + 1, transform, values,
 throwables
 )
 return
 }
 task.addOnCompleteListener { task: Task<*>? ->
 // Record the return value / exception.
 if (task!!.isCanceled) {
 throwables.put(
 detector,
 CancellationException("The task is canceled.")
 )
 } else if (task.isSuccessful) {
 /** ADDED: check on detector type to perform transformation on barcode boundingBox */
 when (detector.detectorType) {
 BARCODE_SCANNER_DETECTOR_TYPE -> {
 val value : List<Barcode> = (task.getResult() as List<*>).let { barcodes ->
 if (barcodes.isEmpty()) emptyList<Barcode>()
 barcodes.map { barcode ->
 (barcode as Barcode).let {
 it.copy(
 boundingBox = it.boundingBox.let { box: Rect? ->
 val rectF = it.boundingBox?.toRectF()
 transform.mapRect(rectF)
 rectF?.toRect()
 }
 )
 }
 }
 }
 values.put(detector, value)
 }
 else -> {
 values.put(detector, task.getResult())
 }
 }
 } else {
 throwables.put(detector, task.exception)
 }
 // Go to the next detector.
 detectRecursively(
 imageProxy, detectorIndex + 1, transform, values,
 throwables
 )
 }
 }
 override fun getDefaultTargetResolution(): Size {
 var size: Size = DEFAULT_SIZE
 for (detector in detectors) {
 val detectorSize = getTargetResolution(detector.detectorType)
 if (detectorSize.height * detectorSize.width
 > size.width * size.height
 ) {
 size = detectorSize
 }
 }
 return size
 }
 private fun getTargetResolution(detectorType: Int): Size {
 return when (detectorType) {
 Detector.TYPE_BARCODE_SCANNING, Detector.TYPE_TEXT_RECOGNITION -> Size(1280, 720)
 else -> DEFAULT_SIZE
 }
 }
 override fun getTargetCoordinateSystem(): Int {
 return targetCoordinateSystem
 }
 override fun updateTransform(matrix: Matrix?) {
 sensorToTarget = if (matrix == null) {
 null
 } else {
 Matrix(matrix)
 }
 }
 class Result(
 private val values: MutableMap<Detector<*>, Any>,
 val timestamp: Long,
 private val throwables: MutableMap<Detector<*>, Throwable?>
 ) {
 /**
 * Get the analysis result for the given ML Kit `Detector`.
 *
 *
 * Returns `null` if the detection is unsuccessful.
 *
 *
 * This method and [.getThrowable] may both return `null`. For example,
 * when a face detector processes a frame successfully and does not detect any faces.
 * However, if [.getThrowable] returns a non-null [Throwable], then this
 * method will always return `null`.
 *
 * @param detector has to be one of the `Detector`s provided in
 * [CustomKitAnalyzer]'s constructor.
 */
 @Suppress("UNCHECKED_CAST")
 fun <T> getValue(detector: Detector<T?>): T? {
 checkDetectorExists(detector)
 return values[detector] as T?
 }
 /**
 * The error returned from the given `Detector`.
 *
 *
 * Returns `null` if the `Detector` finishes without exceptions.
 *
 * @param detector has to be one of the `Detector`s provided in
 * [CustomKitAnalyzer]'s constructor.
 */
 fun getThrowable(detector: Detector<*>): Throwable? {
 checkDetectorExists(detector)
 return throwables[detector]
 }
 private fun checkDetectorExists(detector: Detector<*>) {
 Preconditions.checkArgument(
 values.containsKey(detector) || throwables.containsKey(detector),
 "The detector does not exist"
 )
 }
 }
 companion object {
 private const val TAG = "CustomKitAnalyzer"
 private val DEFAULT_SIZE = Size(480, 360)
 /** ADDED: Detector Type for class BarcodeScanner*/
 private const val BARCODE_SCANNER_DETECTOR_TYPE = 1
 }
}
/**
 * Utility function for barcode copying
 */
fun Barcode.copy(
 format: Int = this.format,
 valueType: Int = this.valueType,
 boundingBox: Rect? = this.boundingBox,
 calendarEvent: Barcode.CalendarEvent? = this.calendarEvent,
 contactInfo: Barcode.ContactInfo? = this.contactInfo,
 driverLicense: Barcode.DriverLicense? = this.driverLicense,
 email: Barcode.Email? = this.email,
 geoPoint: Barcode.GeoPoint? = this.geoPoint,
 phone: Barcode.Phone? = this.phone,
 sms: Barcode.Sms? = this.sms,
 url: Barcode.UrlBookmark? = this.url,
 wifi: Barcode.WiFi? = this.wifi,
 displayValue: String? = this.displayValue,
 rawValue: String? = this.rawValue,
 rawBytes: ByteArray? = this.rawBytes,
 cornerPoints: Array<out Point?>? = this.cornerPoints
) = Barcode(object : BarcodeSource {
 override fun getFormat(): Int = format
 override fun getValueType(): Int = valueType
 override fun getBoundingBox(): Rect? = boundingBox
 override fun getCalendarEvent(): Barcode.CalendarEvent? = calendarEvent
 override fun getContactInfo(): Barcode.ContactInfo? = contactInfo
 override fun getDriverLicense(): Barcode.DriverLicense? = driverLicense
 override fun getEmail(): Barcode.Email? = email
 override fun getGeoPoint(): Barcode.GeoPoint? = geoPoint
 override fun getPhone(): Barcode.Phone? = phone
 override fun getSms(): Barcode.Sms? = sms
 override fun getUrl(): Barcode.UrlBookmark? = url
 override fun getWifi(): Barcode.WiFi? = wifi
 override fun getDisplayValue(): String? = displayValue
 override fun getRawValue(): String? = rawValue
 override fun getRawBytes(): ByteArray? = rawBytes
 override fun getCornerPoints(): Array<out Point?>? = cornerPoints
})

A note on matrix transformations

Like me, you may be interested in the Barcode.boundingBox information.

When not using a PreviewView with CameraController set, alongside CameraController.setImageAnalysisAnalyzer() manual transformations seem required for the Barcode.boundingBox : Rect to display correctly.

The above code includes a transformation performed on any resulting barcode's bounding boxes. This is in imitation of the original process function signature (fun Detector.process(image: Image, rotation: Int, transform: Matrix)), which takes in a transform matrix. The processing function (fun Detector.process(image: InputImage)) takes no such matrix, and Barcode.boundingBox : Rect did not display correctly until applying the calculated analysisToTarget matrix.

The matrix should probably be applied before processing (whether it impacts performance scanning or otherwise I don't know).

I am using targetCoordinateSystem = COORDINATE_SYSTEM_SENSOR, and can't guarantee it'll work as intended with COORDINATE_SYSTEM_ORIGINAL.

Further matrix transformation is required -> use a SurfaceRequest.TransformationInfoListener to optain SurfaceRequest.TransformationInfo, and if using a CameraXviewfinder use the given MutableCoordinateTransformer. The below function transforms a List<Compose.Ui.Rect> to appropriate coordinates:

fun List<Rect>.transformToUiCoords(
 transformationInfo: SurfaceRequest.TransformationInfo?,
 uiToBufferCoordinateTransformer: MutableCoordinateTransformer
): List<Rect> {
 val bufferToUiTransformMatrix = Matrix().apply {
 setFrom(uiToBufferCoordinateTransformer.transformMatrix)
 invert()
 }
 val sensorToBufferTransformMatrix = Matrix().apply {
 transformationInfo?.let {
 setFrom(it.sensorToBufferTransform)
 }
 }
 return this.map { rect ->
 val bufferRect = sensorToBufferTransformMatrix.map(rect)
 val uiRect = bufferToUiTransformMatrix.map(bufferRect)
 uiRect
 }
}