From some documents said OpenMAX has been deprecated for a long time and isn't supported with 64-bit kernels, is it right? If it is, What is the uptodate multimedia platform on RPi?

V4L2 is the mainline Linux kernel API for codec and camera related multimedia, and is where we are heading for those. Codecs are all supported already, as is the ISP for resizing and format conversion.

libcamera is coming in to provide a fully featured (and open source) camera stack.

DRM/KMS, and EGL are the mainline Linux kernel APIs for display side stuff, and again where things are heading.

For the older Pi variants MMAL is there and supported. There is an issue in 64bit land at present with it that will be addressed when time and resource permits.

OpenMAX has been a dead since around 2012. Android used it as most SoC vendors claimed support as part of the tick-box "APIs supported" marketing blurb, but whilst it claimed to be a standard there were significant differences in the interpretation by vendors. It's always been a pain to handle with regard the asynchronous callbacks for event completion, and debugging why transitions never completed was always a nightmare. RIP OMX.
And there is certainly no way I intend to attempt making OMX work in a 64bit environment - it's just too involved and messy.

Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Will the V4L2 H264 decoder be able to decode interlaced video in the (near) future?

Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

So if the OpenMAX API is removed from kernel,which API should I use to operator audio device? Currently I only know use OMX Component to play audio.

italiansc wrote: ↑

Tue Aug 04, 2020 2:55 pm

So if the OpenMAX API is removed from kernel,which API should I use to operator audio device? Currently I only know use OMX Component to play audio.

ALSA - the Linux standard audio library that has been supported on the Pi since about 2015. It also supports I2S and USB sound cards, unlike OMX.

For audio codecs use FFmpeg and the associated libavcodec library - that has far greater codec support than OMX has ever provided, and audio is far lighter on the CPU than video so shifting it to the GPU has minimal gain.

Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

italiansc wrote: ↑

Tue Aug 04, 2020 2:55 pm

So if the OpenMAX API is removed from kernel,which API should I use to operator audio device? Currently I only know use OMX Component to play audio.

You should use the standard linux ALSA components, which is how the desktop does things.

edit: Ninja'd

Software guy, working in the applications team.

So what's going to be the way to make use of the Pis' hardware video decoding capabilities?
I spent some time with ffmpeg/avcodec and wasn't able do do any decent h264 decoding. (I'm happy to be proved wrong, I'm no expert)
With 'hello_video' I can play a 1080p video at full speed using 5-10% of the RPi4's CPU. I wasn't able to get more than 15fps with avcodec and that was at 45% cpu.

I'm just concerned as I haven't seen any examples of any other way of efficiently playing h264 video well.

If you are really leaving OpenMax behind I think it would be a good idea to create some examples showing the alternative ways to get the same results?

RedMarsBlueMoon wrote: ↑

Mon Nov 02, 2020 12:27 am

So what's going to be the way to make use of the Pis' hardware video decoding capabilities?
I spent some time with ffmpeg/avcodec and wasn't able do do any decent h264 decoding. (I'm happy to be proved wrong, I'm no expert)
With 'hello_video' I can play a 1080p video at full speed using 5-10% of the RPi4's CPU. I wasn't able to get more than 15fps with avcodec and that was at 45% cpu.

I'm just concerned as I haven't seen any examples of any other way of efficiently playing h264 video well.

If you are really leaving OpenMax behind I think it would be a good idea to create some examples showing the alternative ways to get the same results?

the h264 hw decode is being migrated over to the v4l api, and the ffmpeg shipped with rpios has been patched to be able to use all hw decode via v4l, and thats also the only way to get hw decode on a 64bit userland, due to complications with how the old api worked

omxplayer and OpenMAX relied on the GPU being able to randomly slap layers onto the screen.
That isn't the way that mainline Linux wants screen composition to go. DRM/KMS is the kernel API that controls display composition.

GStreamer supports KMS natively via kmssink. I've got a 1080p30 Big Buck Bunny running here with that on a Pi4 taking under 3% of any core.
gst-play should work too - I looked at an issue a little while back and got it working. Note that GStreamer knows how to direct audio to IL, and if that isn't supported (eg 64bit OS) it can fail. You really need to direct it to use ALSA.

Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Hi @6by9 that's cool could I please ask what version of gstreamer you are using as I don't seem to have a kmssink available. I built v 1.12 I think it was.
returns nothing and running your line gives a sink not found error.

UPDATE: D'oh its meant to be 'kmssink' and not 'kmsink' of course!
It still doesn't seem to work for me on any of my test video files,

Buster ships with GStreamer 1.14. Latest is 1.16 or so.

KMS supports a single client at a time. At a guess you're running Xserver, in which case that is the one client.
Either boot to the console, or use ctrl-alt-F2 to switch to an alternate console and kmssink will be available.

Whilst running X you have to ask X to be involved in the rendering. Unfortunately the only mechanism it knows about for composition is using GL which hammers performance. eglsink is the one IIRC, or try autovideosink.

And note this is all for Pi3 or 4 running vc4-fkms-v3d or vc4-kms-v3d overlays.
The only place IL isn't going to be supported is on 64bit OSes (ie we're not removing it from 32bit versions), and under those conditions you should be using one of those overlays.

Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

When will the ffmpeg with hw accelerated h264/hevc decoder & encoder be released? the v4l codec api is hard to learn,the documents and examples about it is rare,so I still prefer to use ffmpeg;

Or, how do I add the patches that ffmpeg can support hw video decoder/encoder?