gstreamer: v4l2convert / v4l2h264enc for efficient from RGB(A) to I420 to h264 encoding

Locked

Print view

9 posts • Page 1 of 1

milianw: Posts: 5; Joined: Tue Apr 21, 2020 1:15 pm

gstreamer: v4l2convert / v4l2h264enc for efficient from RGB(A) to I420 to h264 encoding

Tue Apr 21, 2020 1:50 pm

Hey there!

We are trying to optimize a gstreamer pipeline running on a rpi3b+ where, according to gst-shark, the current main bottleneck is a videoconvert element. That one is necessary to convert OpenGL frames in RGBA format to YUV for omxh264enc. This is a simplified example pipeline:

Code: Select all

gst-launch-1.0 gltestsrc \
 ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
 ! gldownload ! queue \
 ! videoconvert \
 ! omxh264enc ! avimux ! filesink location=test.avi

My first approach was trying to get omxh264enc work directly with RGBA frames, see: https://gitlab.freedesktop.org/gstreame ... equests/62

This in turn led me to v4l2convert and v4l2h264enc (see viewtopic.php?t=245852). We have now updated our stack to the following versions:

Code: Select all

linux 4.19.88
gstreamer 1.16.1

When I replace the omxh264enc with v4l2h264enc I run into the first issue:

Code: Select all

gst-launch-1.0 gltestsrc \
 ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
 ! gldownload ! queue \
 ! videoconvert \
 ! v4l2h264enc ! avimux ! filesink location=test.avi
ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: Internal data stream error.
Additional debug info:
../../../../gstreamer-1.16.1/libs/gst/base/gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0:
streaming stopped, reason not-negotiated (-4)

The full output with GST_DEBUG="*:3" is:

Code: Select all

Setting pipeline to PAUSED ...
0:00:00.238494323 1765 0x2361ba0 WARN v4l2 gstv4l2object.c:4196:gst_v4l2_object_probe_caps:<v4l2h264enc0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
Pipeline is PREROLLING ...
0:00:00.242415365 1765 0x23386c0 FIXME default gstutils.c:3981:gst_pad_create_stream_id_internal:<gltestsrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
Got context from element 'gldownloadelement0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0";
0:00:00.264218855 1765 0x23381b0 WARN basetransform gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0> transform could not transform video/x-raw, format=(string)RGBA, width=(int)640, height=(int)480, framerate=(fraction)30/1, interlace-mode=(string)progressive in anything we support
0:00:00.271333542 1765 0x23381b0 WARN basetransform gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0> transform could not transform video/x-raw, format=(string)RGBA, width=(int)640, height=(int)480, framerate=(fraction)30/1, interlace-mode=(string)progressive in anything we support
0:00:00.271435157 1765 0x23381b0 WARN GST_PADS gstpad.c:4231:gst_pad_peer_query:<queue0:src> could not send sticky events
0:00:00.293517917 1765 0x23581b0 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info log:Compiled
0:00:00.294581980 1765 0x23581b0 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info log:Compiled
0:00:00.306531615 1765 0x23581b0 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info log:Compiled
0:00:00.307533698 1765 0x23581b0 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info log:Compiled
0:00:00.342931927 1765 0x23381b0 WARN basetransform gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0> transform could not transform video/x-raw, format=(string)RGBA, width=(int)640, height=(int)480, framerate=(fraction)30/1, interlace-mode=(string)progressive in anything we support
0:00:00.350237917 1765 0x23381b0 WARN basetransform gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0> transform could not transform video/x-raw, format=(string)RGBA, width=(int)640, height=(int)480, framerate=(fraction)30/1, interlace-mode=(string)progressive in anything we support
0:00:00.354808438 1765 0x23386c0 WARN basesrc gstbasesrc.c:3072:gst_base_src_loop:<gltestsrc0> error: Internal data stream error.
0:00:00.355374011 1765 0x23386c0 WARN basesrc gstbasesrc.c:3072:gst_base_src_loop:<gltestsrc0> error: streaming stopped, reason not-negotiated (-4)
0:00:00.356358646 1765 0x23386c0 WARN queue gstqueue.c:988:gst_queue_handle_sink_event:<queue0> error: Internal data stream error.
ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: Internal data stream error.
Additional debug info:
../../../../gstreamer-1.16.1/libs/gst/base/gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
0:00:00.357002709 1765 0x23386c0 WARN queue gstqueue.c:988:gst_queue_handle_sink_event:<queue0> error: streaming stopped, reason not-negotiated (-4)
Freeing pipeline ...

I can manage to workaround this by explicitly letting the videoconvert do the YUV conversion like this:

Code: Select all

gst-launch-1.0 gltestsrc \
 ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
 ! gldownload ! queue \
 ! videoconvert ! "video/x-raw,format=I420" \
 ! v4l2h264enc ! avimux ! filesink location=test.avi

No more errors, but the resulting test.avi file is completely broken. I can play it with cvlc but its all green and garbled...

Now, let's try to stick to omxh264enc and replace videoconvert instead:

Code: Select all

gst-launch-1.0 gltestsrc \
 ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
 ! gldownload ! queue \
 ! v4l2convert \
 ! omxh264enc ! avimux ! filesink location=test.avi
WARNING: erroneous pipeline: could not link queue0 to v4l2convert0

Which is odd, but expected considering the following:

Code: Select all

gst-inspect-1.0 v4l2convert
...
 video/x-raw
 format: { (string)YUY2, (string)YVYU, (string)UYVY, (string)I420, (string)YV12, (string)RGB, (string)BGR, (string)BGR
x, (string)BGRA, (string)RGB16, (string)NV12, (string)NV21 }
 width: [ 1, 32768 ]
 height: [ 1, 32768 ]
 framerate: [ 0/1, 2147483647/1 ]

So, why is RGBA not supported here? Well, let's try to add a glcolorconvert to BGRA, but that fails too:

Code: Select all

gst-launch-1.0 gltestsrc \
 ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
 ! glcolorconvert ! "video/x-raw(memory:GLMemory),format=BGRA" \
 ! gldownload ! queue \
 ! v4l2convert \
 ! omxh264enc ! avimux ! filesink location=test.avi
ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: Internal data stream error.
Additional debug info:
../../../../gstreamer-1.16.1/libs/gst/base/gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn't want to preroll.

I've tried some of the other RGB formats, but none seem to work here. What am I missing? I'm attaching the full :4 GST_DEBUG trace log file too:

trace.log.txt.gz: (9.95 KiB) Downloaded 405 times

Thanks a lot

6by9: Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator; Posts: 18476; Joined: Wed Dec 04, 2013 11:27 am

Re: gstreamer: v4l2convert / v4l2h264enc for efficient from RGB(A) to I420 to h264 encoding

Tue Apr 21, 2020 5:03 pm

v4l2h264enc will accept the same set of formats as v4l2convert as they are both using the ISP hardware block to do the conversion. I mapped through all the formats that I had existing handling for, and RGBA obviously wasn't one of them. Adding it is a turn the handle job, but I'm surprised GL won't produce BGRA if you ask it to. Getting v4l2h264enc to o the conversion is actually more efficient too, as the ISP will get used whether you give it I420 or BGRA.

Sorry, I find interpreting GStreamer failures tricky, so can't really say why it's complaining over your pipeline.

Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

milianw: Posts: 5; Joined: Tue Apr 21, 2020 1:15 pm

Re: gstreamer: v4l2convert / v4l2h264enc for efficient from RGB(A) to I420 to h264 encoding

Tue Apr 21, 2020 5:59 pm

OK, I'll ask on the gstreamer mailing list then for more support on the issues I'm running in there. I believe glcolorconvert does support the RGBA -> BGRA conversion just fine, so I guess it's rather something else...

Anyhow, could you please check whether the colors are correct for you when you run this pipeline:

Code: Select all

gst-launch-1.0 gltestsrc \
 ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
 ! gldownload ! queue \
 ! videoconvert ! "video/x-raw,format=I420" \
 ! v4l2h264enc ! avimux ! filesink location=test.avi

As I said, this gives me a corrupted stream which is mostly green...

ndufresne: Posts: 1; Joined: Thu Apr 23, 2020 7:45 pm

Re: gstreamer: v4l2convert / v4l2h264enc for efficient from RGB(A) to I420 to h264 encoding

Thu Apr 23, 2020 7:50 pm

You should add glcolorconvert element before glupload to let your GPU swizzle the colors to something your encoder supports. RGBA is a very uncommon format outside of GPUs. Be aware the there is no zero-copy support between gldownload and V4L2. Both the download and the copy to V4L2 memory can be expensive. That would need more work to be implemented, possible some work in Mesa also.

6by9: Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator; Posts: 18476; Joined: Wed Dec 04, 2013 11:27 am

Re: gstreamer: v4l2convert / v4l2h264enc for efficient from RGB(A) to I420 to h264 encoding

Fri Apr 24, 2020 10:14 am

Thanks ndufresne - it's nice to have a GStreamer expert pop in here.

Our V4L2 M2M encoder has to do an internal format convert anyway so we're using the ISP hardware block. That can do RGBA as easily as BGRA, but the other infrastructure wasn't there. Dropping the need for an actual convert in there will be beneficial even if it doesn't save the transfers elsewhere.
I need to look at combining the H264 header bytes into the first frame buffer, so will see how easily RGBA support falls out at the same time.

Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

milianw: Posts: 5; Joined: Tue Apr 21, 2020 1:15 pm

Re: gstreamer: v4l2convert / v4l2h264enc for efficient from RGB(A) to I420 to h264 encoding

Fri Apr 24, 2020 10:28 am

Thanks a lot for the input ndufresne, I tried adding the glcolorconvert but to no avail. I even tried explicitly changing to ARGB or BGRA or something like that, but none of that helps with the green video.

I've also cross-posted this to the gstreamer mailing list now, as I would still like to better understand the error messages. They are pretty terse and thus hard for someone like me to grasp.

@6by9: is the h264 header thing you mention maybe related to that? Could it be workarounded somehow?

Generally, thanks for the input so far from you two! It sounds to me as if the current software landscape is thus not really there yet on the RPI3b+ for the task we are trying to do. At least not when using gstreamer, maybe it would work if we rewrite everything by leveraging the OpenMAX directly...

6by9: Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator; Posts: 18476; Joined: Wed Dec 04, 2013 11:27 am

Re: gstreamer: v4l2convert / v4l2h264enc for efficient from RGB(A) to I420 to h264 encoding

Fri Apr 24, 2020 12:48 pm

milianw wrote: ↑
Fri Apr 24, 2020 10:28 am
Thanks a lot for the input ndufresne, I tried adding the glcolorconvert but to no avail. I even tried explicitly changing to ARGB or BGRA or something like that, but none of that helps with the green video.

I've also cross-posted this to the gstreamer mailing list now, as I would still like to better understand the error messages. They are pretty terse and thus hard for someone like me to grasp.

@6by9: is the h264 header thing you mention maybe related to that? Could it be workarounded somehow?

I must confess to having seen green images too, which probably means a dropped reference frame.
It depends on exactly how GStreamer handles the stream. The V4L2 spec isn't clear as to whether header bytes considered to be an independent buffer, or merged with the first frame. I got clarification on that at the Media Summit.
If GStreamer takes the header bytes along as the encoding of the first frame, and drops the "2nd part" which is the actual I-frame, then you will have a corrupt stream with P-frames referencing a non-existent I-frame. Error recovery in codecs will often use a buffer memset to 0 (which is green in YUV space) if they are missing a reference frame. Once the next I-frame appears (typically every 60 frames) the stream should correct itself.

milianw wrote:Generally, thanks for the input so far from you two! It sounds to me as if the current software landscape is thus not really there yet on the RPI3b+ for the task we are trying to do. At least not when using gstreamer, maybe it would work if we rewrite everything by leveraging the OpenMAX directly...

Please don't use IL - it's a right pain in the neck to get working nicely, and actually it won't gain any additional functionality over the V4L2 codec driver. (You actually lose out as IL ends up doing a copy of all the image data from ARM to GPU memory space, so yet another memory bandwidth hit).

Getting data out of 3D subsystems and into "multimedia" subsystems is often non-trivial - they have different optimisation paths, and conversions are needed to get them to match up. We try our best to make it happen.

Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

gvkalra: Posts: 7; Joined: Wed May 15, 2019 10:34 am

Re: gstreamer: v4l2convert / v4l2h264enc for efficient from RGB(A) to I420 to h264 encoding

Wed Apr 29, 2020 7:21 pm

milianw wrote: ↑
Tue Apr 21, 2020 5:59 pm
Anyhow, could you please check whether the colors are correct for you when you run this pipeline:
Code: Select all
gst-launch-1.0 gltestsrc \
 ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
 ! gldownload ! queue \
 ! videoconvert ! "video/x-raw,format=I420" \
 ! v4l2h264enc ! avimux ! filesink location=test.avi
As I said, this gives me a corrupted stream which is mostly green...

I spinned-up a fresh installation of Raspbian buster:

Code: Select all

pi@raspberrypi:~ $ lsb_release -a
No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 10 (buster)
Release:	10
Codename:	buster

After having installed the necessary gstreamer-1.0 packages from apt, I tried your pipeline:

Code: Select all

pi@raspberrypi:~ $ gst-launch-1.0 gltestsrc \
> ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
> ! gldownload ! queue \
> ! videoconvert ! "video/x-raw,format=I420" \
> ! v4l2h264enc ! avimux ! filesink location=test.avi
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'gldownloadelement0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayGBM\)\ gldisplaygbm0";
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:02:26.656833259
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Code: Select all

pi@raspberrypi:~ $ cvlc test.avi 
VLC media player 3.0.8 Vetinari (revision 3.0.8-0-gf350b6b5a7)
[01bdc3a8] vlcpulse audio output error: PulseAudio server connection failure: Connection refused
[01bdfbc0] main interface error: no suitable interface module
[01b626d0] main libvlc error: interface "globalhotkeys,none" initialization failed
[01bdfbc0] dummy interface: using the dummy interface module...
[72801f78] avi demux error: no key frame set for track 0
[726332d0] mmal_codec decoder: VCSM init succeeded: CMA
[726332d0] main decoder error: buffer deadlock prevented
[72a04098] gles2 generic error: parent window not available
[72a0e260] mmal_xsplitter vout display error: Failed to open Xsplitter:opengles2 module
[72a04098] xcb generic error: window not available
[72a0e260] mmal_xsplitter vout display error: Failed to open Xsplitter:xcb_x11 module

It looks okay to me (test.avi)

Code: Select all

pi@raspberrypi:~ $ top
top - 20:05:02 up 1 min, 4 users, load average: 1.85, 0.82, 0.31
Tasks: 151 total, 1 running, 150 sleeping, 0 stopped, 0 zombie
%Cpu(s): 23.0 us, 4.9 sy, 0.0 ni, 72.0 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 874.0 total, 471.4 free, 192.3 used, 210.3 buff/cache
MiB Swap: 100.0 total, 100.0 free, 0.0 used. 621.4 avail Mem 
 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
 1000 pi 20 0 201836 52976 27848 S 109.3 5.9 0:17.05 gst-launch-1.0 
 74 root 1 -19 0 0 0 S 0.7 0.0 0:00.09 vchiq-slot/0 
 1053 pi 20 0 7756 2732 2232 R 0.7 0.3 0:00.04 top 
 10 root 20 0 0 0 0 I 0.3 0.0 0:00.08 rcu_sched 
 169 root 20 0 0 0 0 D 0.3 0.0 0:00.10 ft5406 
 217 root 20 0 0 0 0 I 0.3 0.0 0:00.07 kworker/3:3-events 
 455 root 20 0 149516 48380 32736 S 0.3 5.4 0:01.34 Xorg

Attachments

WhatsApp Image 2020年04月30日 at 04.12.36.jpeg: WhatsApp Image 2020年04月30日 at 04.12.36.jpeg (237.34 KiB) Viewed 34984 times

milianw: Posts: 5; Joined: Tue Apr 21, 2020 1:15 pm

Re: gstreamer: v4l2convert / v4l2h264enc for efficient from RGB(A) to I420 to h264 encoding

Thu Apr 30, 2020 7:03 am

Interesting, then maybe the v4l2h264enc issue with the green frames depends on load and is thus not always going to be visible.

But the load you are seeing (>1core) is quite a lot. With v4l2convert not being a feasible path forward, it seems like we'll have to find a different solution to our problem.

Thanks everyone

Locked

Print view

9 posts • Page 1 of 1

Return to "OpenMAX"

Jump to