I would start with the name (both libMP & MediaPipe): not great، ex. of some existing projects using it:
https://github.com/gpudirect/libmp
https://man.freebsd.org/cgi/man.cgi?query=libmp&sektion=3&format=html
https://github.com/google-ai-edge/mediapipe

Also, shouldn't this be hosted as an external RTOS-agnostic library, like libmpix & LVGL, it would see wider adoption that way in my opinion & would have better APIs.

I would start with the name (both libMP & MediaPipe): not great، ex. of some existing projects using it:
https://github.com/gpudirect/libmp
https://man.freebsd.org/cgi/man.cgi?query=libmp&sektion=3&format=html
https://github.com/google-ai-edge/mediapipe

Indeed, these are existing projects which have the same name with libMP. In fact, we tried to change the name several times, it's difficult to have an intuitive name which does not overlap with the existing ... What about libMPL, I do not see this elsewhere ? Do you know if we need to change also the prefix (mp_) in the code when changing the project name ?

Also, shouldn't this be hosted as an external RTOS-agnostic library, like libmpix & LVGL, it would see wider adoption that way in my opinion & would have better APIs.

I think it's a bit different compared to libmpix & LVGL. AFAIK, libmpix is mainly about math, algorithms (color / format conversions, etc.) which is rather OS-agnostic. LVGL exists nearly at the same time with Zephyr and has it own life-cycle outside Zephyr. Moreover, LVGL has its own eco-system and does not interact much with the OS, except when it comes to touch (input) and HW accelerators (such as PxP, VGLite, but for this, LVGL calls directly to the low level drivers and bypasses Zephyr subsystems and drivers. About this, I don't know how it works when we want to use LVGL and Zephyr stuffs on the PxP in the same application, does it lead to conflicts because Zephyr stuffs will pass by the video subsystem through the PxP Zephyr driver to the PxP low level driver while LVGL stuffs will pass directly to the PxP low level driver ? ...). So, to port LVGL to Zephyr, we need mainly an OSA and some glue codes.

BTW, libMP used (and will use) heavily Zephyr mechanismes such as devicetree, iterable section, work queue, rtio, etc. to optimize the implementation. If we make an OS-agnostic version, we need to create something equivalent or the implementation cannot be optimized. The media components (plugins, elements) in libMP interact deeply with the OS, they calls directly to the (Zephyr) video, audio, display, vision, ... subsystems. So even in a generic libMP version, these components need to be created separately for each OS. And to support FreeRTOS, as an example, where there are no such subsystems we need to create all of them (kind of a HAL layer and need to reproduce the APIs a bit like in Zephyr).

Another reason is, as an external module, libMP is required to have its own life-cycle outside the Zephyr Project, that is, reside in its own repository, and have its own contribution, testing and maintenance workflow and release process. We need to do integration into Zephyr regularly (like LVGL) and review all code from contributors (that may come from many different domains : video, audio, display, vision, NPU, etc.) where we don't have enough resource to do that.

Looking that such a unified multimedia framework does not exist yet in Zephyr (there are some frameworks for audio such as Maestro, but when integrated into Zephyr it bypasses all Zephyr subsystems, so not a real integration), making it inside Zephyr, we expect much more contribution and helps from the Zephyr community and benefit Zephyr infrastructure (the current code base is just an initiative).

So, IMHO, if we support FreeRTOS in the future, we could port it or maintain two versions where the generic version may not be optimized and the Zephyr version may grow much faster and has its own development cycle.

Also we need to understand if this is a Zephyr Subsystem or a Zephyr Library.

Also we need to understand if this is a Zephyr Subsystem or a Zephyr Library.

It seems to me that it's a Zephyr library (?)

For this message, I only look at the content of lib/mp/src/core: the framework source itelf.

It seems like there is some RTOS abstraction layer, which needs to stay if this is not meant as Zephyr-first/only implementation:

• mp_bus.c -> Zbus
• mp_event.c -> Zbus / Events
• mp_messages.c -> Zbus
• mp_task.c -> k_thread_...() (maybe no wrapper needed if Zephyr only target)
• mp_pixel_format.h -> four character codes
• mp_plugin.c -> SYS_INIT() in plugins (does not work if plugins need to support non-Zephyr)
• mp_buffer.c -> net_buf / RTIO buffers / other
• mp_utils.c. -> sys/utils

And then a very small core on top of it, bringing the bulk of what a media subsystem would need to do in an RTOS.
If I got it right:

• mp_value.c, mp_structure.c: A generic configuration library
• mp_bin.c, mp_element.c, mp_element_factory.c: A pipeline configuration API
• mp_pipeline.c, mp_pad.c, mp_src.c, mp_sink.c: A pipeline runtime API
• mp_caps.c, mp_property.h, mp_object.c, mp_query.c, : A pipeline control API

So +1 to try to reduce the number of elements to integrate and abstraction layers:

• Vendor-specific solution: Vendor SDK > Vendor library > Application
• External library on Zephyr: Vendor HAL > Zephyr > RTOS abstraction layer > library > application
• Internal library on Zephyr: Vendor HAL > Zephyr > Application

Some "complex" or "multi-component" camera/video hardware is arriving to Zephyr:

• i.MXRT1170: MIPI + scaler + display (maybe more evolved hardware coming)
• STM32N6: MIPI + debayer + ISP + scaler + encoder + display
• ESP32-P4: MIPI + debayer + ISP + scaler + encoder + display
• Renesas RA8D2: MIPI + ISP + scaler + display
• MPUs converted from Linux to Zephyr (like STM32MP1 or SG2000)
• ...

Depending on the hardware, a different application has to be written (currently managed with a growing number of #ifdef), unless there exists a framework to turn this variability into configuration.

In that sense, libMP can also be seen as an essential part of video hardware integration as it enables writing the basic video samples without hundreads of lines of copy-pasta boilerplate.

For instance, here is Zephyr implementation of libMP's element->srcpads locally inside the UVC sample:

This encourages adding a dependency from Zephyr video samples to libMP, whichever way it is integrated.

Thanks for the comment.

It seems like there is some RTOS abstraction layer, which needs to stay if this is not meant as Zephyr-first/only implementation:

In fact, these are not RTOS abstraction layer but the "components" that we built to use in libMP. But you are right, there are things that we could (change /)move to other places to lighten the library.

mp_bus.c -> Zbus
mp_messages.c -> Zbus

The "bus" and "message" concept in libMP are much lighter than Zbus. Basically it's just a FIFO containing messages from the pipeline sent to the application (one way) so I think using Zbus is a bit overkill.

mp_event.c -> Zbus / Events

Event in libMP is different from the generic event mechanism and event in Zephyr. As seen in the code, it's simply a structure that contains a pointer to a data structure. There is no mechanism for "listening" or "broadcasting" the event. Element sends an event to downstream or upstream by simply putting it in the function parameters, and the element can handle the event or propagate it but this is implemented inside the element itself.

mp_event and mp_query are nearly the same and should be refactored (will do).

mp_task.c -> k_thread_...() (maybe no wrapper needed if Zephyr only target)

Actually we use k_thread and just refactored into functions to not to duplicate code. Task will be extended more in the future.

mp_pixel_format.h -> four character codes

mp_pixel_format are just enums to unify formats from different domains (video, display, vision. etc.) so that they can understand each other. So an enum is sufficient, I don't see why we need a FOURCC ... and there are some formats (in display) that don't have FOURCC.

mp_plugin.c -> SYS_INIT() in plugins (does not work if plugins need to support non-Zephyr)

Yes, that's right. Instead of calling mp_init() in each application. libMP can be initialized with SYS_INIT(). I will do that. So, by this, it turns out that libMP should be a subsys than a lib.

mp_buffer.c -> net_buf / RTIO buffers / other

Currently libMP buffer pool is just an array of buffer structures to map to the real data buffers comming from each subsystem, no FIFO, no handling mechanism required (it's already done differently in each subsystem, e.g. video subsystem already used RTIO - ongoing work), element push buffer to downstream one by one after processing it. So, I am not sure to be able to use RTIO for this but I thought of that. Will rethink about this when we finished switching to RTIO for video subsystem.

mp_utils.c. -> sys/utils

That's right. This can be taken out and upstream to sys/utils.

mp_value.c, mp_structure.c: A generic configuration library

That's right too. These can be taken out from libMP. But currently I don't know where to put it in Zephyr.

• mp_value are wrappers to support handling value of primitive and non-primitive (range, list) types, doing comparison and intersection operators on them.
• mp_structure is a generic abstracion for a dynamic data structure built on top of mp_value, kind of {field, value} pair which can be appended into a caps structure.

Both are used for caps and query. Basically they are generic and can be used outside libMP but it's hard to find another usage than this one.

Thank you for walking through these points, this helps estimating the overlap with Zephyr features and figure out how to reuse existing Zephyr code to lighten libMP, and where it is not useful/possible to do so.

This could act as integration layer to all of these?

• Image/Audio input drivers (MIPI, DVP, PDM, I2S...)
• Image/Audio output drivers (displays, speakers, I2S...)
• Bluetooth Audio (LEA/auracast?)
• USB (Audio UAC2, Video UVC, both host/device)
• Networking (simple TCP capture, libsrtp support is coming, HTTP-based streaming)
• Storage (recording, playback)
• NPU
• Container formats (mkv, lc3, mpeg-ts, mp3, ogg, opus...)
• Processing (i.e. echo cancellation library, color tuning)
• SOF (?)

Maybe even sensors: combine temperature data with an audio recording of engine noise and send both to an NPU.

Quality Gate Passed Quality Gate passed

Issues
61 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.1% Duplication on New Code

See analysis details on SonarQube Cloud

Fixed parts of Compliance and SonarQubec failures.

This could act as integration layer to all of these?

It's more like an application layer, yes, whenever we can add a plugin / element for these, it will help.

Networking (simple TCP capture, libsrtp support is coming, HTTP-based streaming)

It seems we need an rtph264pay element.

Storage (recording, playback)

It seems we need kind of filesrc element

NPU

For NPU, the problem is we don't have a subsystem. So, to support them in libMP, it seems we need to pass by low level drivers and need to create custom elements for each vendor ...

Container formats (mkv, lc3, mpeg-ts, mp3, ogg, opus...)

Will need parsers elements like : matroskademux, tsdemux, h265parse, etc.

SOF (?)

I am not familiar with SOF but it seems audio topology is also part of SOF, so not sure is there any overlapping with libMP.