We use some essential cookies to make our website work.

We use optional cookies, as detailed in our cookie policy, to remember your settings and understand how you use our website.

Gavinmc42
Posts: 8346
Joined: Wed Aug 28, 2013 3:31 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Mon Oct 18, 2021 6:09 am

https://github.com/librerpi/rpi-open-fi ... ipeline.md
And the other files in that directory

Code: Select all

VC6 pi's have 5 pixelvalves:
PV0 can drive DSI0/DPI
PV1 can drive DSI1/SMI
PV2 can drive HDMI0
PV3 can drive VEC
PV4 can drive HDMI1
That tends to indicate a Pi4 can do two HDMI and composite at the same time?
I was thinking a small PAL monitor would be handy for a debug output.
DPI could do it too but not an easy plugin and debug feature.
Are you sure there are 5 Pixelvalves?
5 screens on a CM4.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

cleverca22
Posts: 9593
Joined: Sat Aug 18, 2012 2:33 pm

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Mon Oct 18, 2021 6:16 am

Gavinmc42 wrote:
Mon Oct 18, 2021 6:09 am
That tends to indicate a Pi4 can do two HDMI and composite at the same time?
I was thinking a small PAL monitor would be handy for a debug output.
but you have an added complication there
the PAL/NTSC mode, needs a 108mhz clock, so one of the PLL's needs to run at an integer multiple of 108mhz

the HDMI also needs a pixel clock (monitor/resolution dependant), so one of the PLL's needs to run at an integer multiple of that
add in a second HDMI of a different resolution, and now you either tie up 3 PLL's or you have to start doing fancy math to try and solve the whole problem

the current linux drivers just dont try to solve that, and refuse to even try
and the closed-source firmware drivers can only drive 2 displays at once due to their design, so you cant even configure it to do 3
Gavinmc42 wrote:
Mon Oct 18, 2021 6:09 am
Are you sure there are 5 Pixelvalves?
5 screens on a CM4.
5 pixel valves, but only 3 channels on the HVS that generates image data
so you must select a max of 3 PV's from the set of 5

with more work on the linux drivers, it may be possible to solve that

Gavinmc42
Posts: 8346
Joined: Wed Aug 28, 2013 3:31 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Mon Oct 18, 2021 6:40 am

5 pixel valves, but only 3 channels on the HVS that generates image data
so you must select a max of 3 PV's from the set of 5

with more work on the linux drivers, it may be possible to solve that
So possible for 3 displays in baremetal?
The VC4's had HDMI or VEC, not both at the same time.
Somehow I thought that still applied for VC6.
Two HDMI the same size use the same PLL, use another one for VEC?
How many PLLs does the Pi4 have? Time to dig out that BCM2711 datasheet again.
A,C,D, HDMI auxiliary, same as BCM2835?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

cleverca22
Posts: 9593
Joined: Sat Aug 18, 2012 2:33 pm

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Mon Oct 18, 2021 6:54 am

Gavinmc42 wrote:
Mon Oct 18, 2021 6:40 am
Two HDMI the same size use the same PLL, use another one for VEC?
yeah, thats one shortcut you can definitely use

https://elinux.org/The_Undocumented_Pi#Clocks

this doc will also be of use
but it hasnt been updated for the vast changes the bcm2711 has done

you can also run "cat /sys/kernel/debug/clk/clk_summary" to see the current clock tree
but that wont show the possible muxing, just the currently chosen routing

swooby
Posts: 3
Joined: Wed Sep 29, 2021 7:52 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Mon Oct 18, 2021 6:57 am

Y'all are talking over my head.

I'm still hung up on the:
Bleeding edge, pack bandaids.
This is just video playback, so I guess I need to reset my expectations of what bleeding edge may mean on these devices.

I would have expected public domain code on how to efficiently render video would have been one of the first obvious things to achieve on these, perhaps even working with the pre-release development of the boards to have it evaluated before release and ready for release.

I am also evaluating using a Nvidia Jetson Nano 2GB: even though it is arguably cheaper than a Pi4, its main advantage is that it has a better GPU (cuda based), but the Pi4's advantage is that its hardware and community support is more ubiquitous. To me it is really just a race to see which one I can find or implement fast free video playback on in my own binary.

This is just a hobby of mine on the side, so my progress absorbing from this firehose may be slow.

Gavinmc42
Posts: 8346
Joined: Wed Aug 28, 2013 3:31 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Mon Oct 18, 2021 8:02 am

I would have expected public domain code on how to efficiently render video would have been one of the first obvious things to achieve on these,
Under Linux it was done and is done, even more ways now as more video players get recompiled to run on ARM cpus.
You can even compile them yourself.
Most are just a "git clone xxx, cd xx, mkdir build, cd build, make".
Most of those Linux video players will use ffmpeg.
DIY compile the latest version, example https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu

"Sudo apt-get install ffmpeg" is the usual way to install.
ffmpeg is a handy thing to install for video file format conversions.
Most people use ffmpeg because it has so many encoders/decoders built in.

If you just want to use one format of video file and write your own decoder, then that is harder.
Easy in software, harder accelerated due to VC6 secret sauce stuff.
The Raspberry Pi engineers are offloading some/most of the latest codecs to the ARM cores as they are faster than the VC6 video codec hardware. That makes them opensource too.

But you sounded like you wanted to do it without an OS.
To do it that way you need to check out the Baremetal forum posts.
Those are big firehoses :lol:

I prefer to use someone else's low level libs and just write the high level application user interface.
A MP3 decoder would be my first one to try DIY as WAV and H264 already work.
WAV is not compressed so MP3 audio makes sense.

A combo audio/video decoder like MP4 needs a wrapper layer.
For Pi4's I would just do AV1, as it seems to be winning the codec skirmishes.
But to optimize it for simple CPU's like Pi's, assembly language looks like it gets used in dav1d, another firehose ;)

You could do an old format for practice but they will be low resolution.
It all sounds complex because it is ;)

If you want the easy way to make a Player binary that runs on a Pi in Linux then using Lazarus/Free Pascal with the VLC lib could do it.
https://lazplanet.blogspot.com/2018/01/ ... er-in.html
I think it should be even easier on the latest Laz/fpc which works ok on my Pi400.
Have not tried that myself yet.

Non stop learning with Pi's.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 35174
Joined: Sat Jul 30, 2011 7:41 pm

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Mon Oct 18, 2021 8:26 am

We provide standard Linux API's to access the HW encoders (V4L2 etc), both for H264 and HEVC(H265). The H264 encoder register interface is closed, but the HEVC one is, I believe, open. However I don't recommend accessing the HW directly - use the official API's as they will not change over HW revisions. We no longer recommend use of MMAL, the legacy way of accessing the codec HW, as that is being deprecated as we move to a more standard system.

So, checkout V4L2 and gstreamer on Pi for more details.
Software guy, working in the applications team.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 18476
Joined: Wed Dec 04, 2013 11:27 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Mon Oct 18, 2021 2:55 pm

Gavinmc42 wrote:
Mon Oct 18, 2021 6:40 am
5 pixel valves, but only 3 channels on the HVS that generates image data
so you must select a max of 3 PV's from the set of 5

with more work on the linux drivers, it may be possible to solve that
So possible for 3 displays in baremetal?
The VC4's had HDMI or VEC, not both at the same time.
Somehow I thought that still applied for VC6.
Two HDMI the same size use the same PLL, use another one for VEC?
How many PLLs does the Pi4 have? Time to dig out that BCM2711 datasheet again.
A,C,D, HDMI auxiliary, same as BCM2835?
3 displays simultaneously works now in vc4-kms-v3d. I've had:
- dual HDMI + single DSI
- single HDMI + dual DSI
- dual HDMI + DPI
all working.

DSI0 and DPI are mutually exclusive (they share PV0), but otherwise it's a free choice.

Each HDMI output has a separate PLL. Locking the VEC to 108MHz means compromises on other peripherals. Some of those compromises have been worked around by reconfiguring the clock tree, so they aren't as bad as they used to be.
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Gavinmc42
Posts: 8346
Joined: Wed Aug 28, 2013 3:31 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Tue Oct 19, 2021 12:51 am

Thanks 6by9 for the display options confirmation, I was still stuck in VC4 perception land :oops:

What is SMI video mode?
Is that the old 68/80 bus type monochrome TN LCD displays?
Some of those are pretty good in daylight.
Outdoor Cyberdeck?

Is there any links re PLL peripheral compromises?
Uart, i2s could be an issue, clock tolerance for i2c, SPI not so much.

Thanks also for Gstreamer suggestion, interesting stuff there.
It led to the dav1d decoder which just compiled on a Pi400.
No idea how to use it yet but it was painless to get it compiled, that's a good sign.

Turns out the Lazarus/fpc VLC example works too, just had it playing the full BigBuckBunny :D
Scales to window size without glitching, HVS is working.

Basically V4L2 for encoding, Gstreamer for converting/decoding?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

cleverca22
Posts: 9593
Joined: Sat Aug 18, 2012 2:33 pm

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Tue Oct 19, 2021 12:56 am

Gavinmc42 wrote:
Tue Oct 19, 2021 12:51 am
What is SMI video mode?
Is that the old 68/80 bus type monochrome TN LCD displays?
Some of those are pretty good in daylight.
Outdoor Cyberdeck?
got some links to those screen datasheets?
ive only used smi in the manual/bus mode, and not as a video output, and i'm interested in how using it as a display would be different

Gavinmc42
Posts: 8346
Joined: Wed Aug 28, 2013 3:31 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Tue Oct 19, 2021 1:27 am

got some links to those screen datasheets?
A search found a few 320x240, 640x200 are the bigger sizes.

Digikey have some Newhaven Graphics LCDs that might have that interface.
Only 320x240 https://www.newhavendisplay.com/specs/N ... VZ-3VR.pdf
A search for "8bit parallel" gives a bunch of graphical mono displays that could be this interface.

Any mono LCD graphical LCD pulled out of old equipment could be recycled?
Won't be video rates but good enough for outdoor text/typing/editing.
Epson HX-20, Tandy 100, Toshiba T1100 type of computer.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

cleverca22
Posts: 9593
Joined: Sat Aug 18, 2012 2:33 pm

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Tue Oct 19, 2021 1:29 am

Gavinmc42 wrote:
Tue Oct 19, 2021 1:27 am
Won't be video rates but good enough for outdoor text/typing/editing.
that leaves me wondering, does the rpi just blindly update the entire screen at the configured vsync, or is it diffing?

Gavinmc42
Posts: 8346
Joined: Wed Aug 28, 2013 3:31 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Tue Oct 19, 2021 1:52 am

that leaves me wondering, does the rpi just blindly update the entire screen at the configured vsync, or is it diffing?
Digging into the dusty part of my memory, I think it depends on the controller chip/driver.
Some required it all external, some had RAM on them.
A LCD with controller normally had RAM.
Just driver LCDs needed an external controller with RAM.

As to what the old Linux SMI drivers have, that will need some source code diving.
These were not fast displays compared to todays LCDs, might even be able to bitbang them.
Pre TFT LCDs, the old days of mono LCDs.
Are the drivers still in Linux, the 44780 character LCD driver is, not many others, 128x64.
https://github.com/torvalds/linux/tree/ ... auxdisplay
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 18476
Joined: Wed Dec 04, 2013 11:27 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Tue Oct 19, 2021 7:34 pm

Anything that comes out of the HVS will be full frames.
Whilst the hardware docs describe a route to send data from the HVS/PV to SMI, I have no details about how to use, and unless management find a use case then I'm not intending on investigating it.
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

cleverca22
Posts: 9593
Joined: Sat Aug 18, 2012 2:33 pm

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Tue Oct 19, 2021 7:38 pm

i should try turning it on some time and see what comes out of the gpio

should be a fun experiment!

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8471
Joined: Wed Aug 17, 2011 7:41 pm

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Tue Oct 19, 2021 8:03 pm

Gavinmc42 wrote:
Mon Oct 18, 2021 2:13 am
Been a while since I checked, AV1 looks to be the Codec many are using/going to use.
Is dav1d the fastest decoder?
https://code.videolan.org/videolan/dav1d
Optimized C and assembler for the various Arm chips Pi use :D

Will it compile on a Pi?
Yes, dav1d is the best av1 decoder. It's heavily optimised for arm (and x86).
It builds and runs on Pi (e.g. it's available with LibreELEC).
Handles 720p okay on a Pi4. 1080p is unlikely without heroic effort.

Gavinmc42
Posts: 8346
Joined: Wed Aug 28, 2013 3:31 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Tue Oct 19, 2021 11:52 pm

i should try turning it on some time and see what comes out of the gpio

should be a fun experiment!
Yep, if it comes out of the HVS as full frame then it is probably going to be a LCD driver interface not controller with RAM.
Always been curious about the SMI stuff.
Handles 720p okay on a Pi4. 1080p is unlikely without heroic effort.
Dav1d with 720p videos is pretty good, got it to compile on my Pi400.
https://test-videos.co.uk/
All my monitors are less than 1920x1080, going to need a TV to test the big AV1 files.

So for 4K vids on Pi's what is currently the best format, H.264 or HVEC(H.265) and which is the best decoder?
Just curious, I don't have a 4K screen anyway.
Hmm, going off to read the LibreELEC info.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

cleverca22
Posts: 9593
Joined: Sat Aug 18, 2012 2:33 pm

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Wed Oct 20, 2021 1:37 am

Gavinmc42 wrote:
Tue Oct 19, 2021 11:52 pm
Always been curious about the SMI stuff.
https://iosoft.blog/2020/07/16/raspberry-pi-smi/
https://github.com/librerpi/rpi-tools/b ... smi-test.c

this has info on how the SMI works (in non-video mode) and an example of driving it from linux
Gavinmc42 wrote:
Tue Oct 19, 2021 11:52 pm
So for 4K vids on Pi's what is currently the best format, H.264 or HVEC(H.265) and which is the best decoder?
i believe the h264 decoder in the bcm2711 is limited to 1080
so for 4k content, you must use h265

Gavinmc42
Posts: 8346
Joined: Wed Aug 28, 2013 3:31 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Wed Oct 20, 2021 3:06 am

i believe the h264 decoder in the bcm2711 is limited to 1080
so for 4k content, you must use h265
So these would be suitable test files.
https://jell.yfish.us/
4K 120-400Mbs yikes

I guess the Pi4 h265 4K performance is using that extra h265 hardware block.
Not sure if it could be used to boost dav1d AV1 decoding.
How much more can the Pi4 be pushed for decoding?

Maybe the Pi5 will have an AV1 hardware decoder, newest chips seem to be getting that assist.
Wonder what the difference is between HEVC and AV1 hardware decoding.
OK, that was an interesting learning search.

I will stick with 720p software decoded AV1s on Pi4s ;)
That is doable in baremetal but it is not accelerated unless you consider NEON an accelerator.
Not sure how to do H265 in baremetal yet.
FFMPEG can convert any vids to MP4/AV1 files.

Considering I could not pick the difference between a 360p 1MB and 720p 30MB BBB I won't be a good judge.
Seen people complain about their 4K vids on Pi's, amazing they work at all.
Must be easier to spot artifacts on 4K screens.

SMI 9 and 18bit mode remind me of a dusty memory that's buried at the moment.
Seen that SMI nand and IDE interface before, SMI video is new to me.
Googoo search - HX8357 also has 9 and 18 bit 262K colour mode?
Breakout board in 8 bit mode, should be faster than SPI, is it SMI mode?
https://learn.adafruit.com/adafruit-3-5 ... ut/pinouts
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

cleverca22
Posts: 9593
Joined: Sat Aug 18, 2012 2:33 pm

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Wed Oct 20, 2021 4:00 am

Gavinmc42 wrote:
Wed Oct 20, 2021 3:06 am
SMI 9 and 18bit mode remind me of a dusty memory that's buried at the moment.
Seen that SMI nand and IDE interface before, SMI video is new to me.
Googoo search - HX8357 also has 9 and 18 bit 262K colour mode?
Breakout board in 8 bit mode, should be faster than SPI, is it SMI mode?
https://learn.adafruit.com/adafruit-3-5 ... ut/pinouts
the silkscreen says 8080, and the SMI has both 8080 and 6800 modes, so that definitely looks viable

Gavinmc42
Posts: 8346
Joined: Wed Aug 28, 2013 3:31 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Wed Oct 20, 2021 7:06 am

Didn't really understand the implications at the time.
https://www.phoronix.com/scan.php?page= ... ional-Spec.
When Vulkan 1.2.x comes to Pi's will it have the codecs?

Vulkan on Pi with H264/H265/AV1 - for some reason Bladerunner popped into my head.
A 3D game with all those videos playing everywhere.
Won't even need 4K, lots of 426 x 240 30fps would be fine.

VR version?
Speaking of which, what is the state of play for AR/VR on Pi's?
Got two HDMI outputs on a Pi4. 2 HDMI and 2 x DSI on a CM4.

For a VR headset, small highest res retina displays will be best.
Waveshare - 5.5" 1920x1080 Amoled, 4.3" 800x480 or use smartphone?
Hmm 8.8" 1920 x 480, cute, shiny.
HDMI 3.2" 800x480, easier optics?

Waveshare is one of those places you take a shopping trolley to.
"I'll have 1 of those, a couple of these, half dozen of them, need that..." :lol:
Getting distracted, again.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 18476
Joined: Wed Dec 04, 2013 11:27 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Wed Oct 20, 2021 10:43 am

Gavinmc42 wrote:
Wed Oct 20, 2021 7:06 am
Didn't really understand the implications at the time.
https://www.phoronix.com/scan.php?page= ... ional-Spec.
When Vulkan 1.2.x comes to Pi's will it have the codecs?
Not as far as I know, unless they write a wrapper over the top of V4L2 stateful or stateless codec APIs.
This is the annoyance that there are multiple APIs, but this appears to add nothing extra other than potentially cross-platform support.
As stated in the article
With Vulkan Video 1.0 and when the expanded codec support is here we stand good chances of this hopefully becoming the dominant interface for GPU-accelerated video encode/decode... Well, hopefully it takes off better than OpenMAX.
Khronos are notorious for writing bad specs, hence the lack of interoperability between difference OpenMAX implementations and slight variations in how they had to be driven. GL and GLES seem to be the exceptions.
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Gavinmc42
Posts: 8346
Joined: Wed Aug 28, 2013 3:31 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Thu Oct 21, 2021 12:05 am

Not as far as I know, unless they write a wrapper over the top of V4L2 stateful or stateless codec APIs.
Guess I better learn about V4L2 now.

I only have two 1280x1024 monitors but I stretched the Browser and VLC across both screens and was able to play 1080 AV1s near actual size on my Pi400.
https://test-videos.co.uk/bigbuckbunny/mp4-av1
Some magic involved in the HVS to scale that window to both HDMI's?
There were some issues so it was probably borderline but not bad.

So was Chromium and VLC using the Pi's extra hardware for decoding or pure software with NEON?
I assume Chromium is using libaom which might have been improved recently.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

cleverca22
Posts: 9593
Joined: Sat Aug 18, 2012 2:33 pm

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Thu Oct 21, 2021 12:26 am

Gavinmc42 wrote:
Thu Oct 21, 2021 12:05 am
Some magic involved in the HVS to scale that window to both HDMI's?
the HVS has a seperate list of images to render, for each output

if you want to stretch an image over 2 outputs, you would use the proper stride, but a reduced width, so it only copied $width pixels out of each line of $stride pixels
you will also offset one of them by $off pixels, so things start at say the middle of the scanline for each line
in theory, x11/linux can just do all of the above, and it will just work

but i suspect the DRM api in linux doesnt allow spreading a framebuffer over 2 outputs?
so it might be converting your frame into a 3d texture, and then rendering 2 scenes, one per monitor?

Gavinmc42
Posts: 8346
Joined: Wed Aug 28, 2013 3:31 am

Re: All about accelerated video on the Raspberry Pi (check my notes)?

Thu Oct 21, 2021 1:05 am

but i suspect the DRM api in linux doesnt allow spreading a framebuffer over 2 outputs?
so it might be converting your frame into a 3d texture, and then rendering 2 scenes, one per monitor?
Yep, like I said "magic".
Play half height 4K vids?
I don't have time to watch 4K or low res movies, to busy learning how the Pi's can do it at all.
Hmm, a USB DVD player on the shopping list, purely for testing of course ;)

I don't remember any portable Pi based DVD players.
Ah, MPEG2 and license file, that's why I am clueless re DVD.
As of February 14 2020, MPEG-2 Patents have expired worldwide, with the exception of only Malaysia.
Does that mean Pi's can now play MPEG2s without license?
That codec is old, low res these days, a Zero should be able to play it.
https://github.com/chetbox/raspberry-pi-dvd-player
Non stop learning with Pi's.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

Return to "Graphics programming"

AltStyle によって変換されたページ (->オリジナル) /