Seeking insight about the framebuffer lifecycle
I'm currently looking into slightly modifying my DRM/KMS based playback engine while trying to reduce memory usage of frame buffers. I'm having some trouble understanding how long to wait before I can safely drmModeRmFB a framebuffer after removing it from a CRTC.
I'm currently playing around with using OUT_FENCE_PTR for each CRTC instead of using DRM_MODE_PAGE_FLIP_EVENT. The documentation says:
I'm currently playing around with using OUT_FENCE_PTR for each CRTC instead of using DRM_MODE_PAGE_FLIP_EVENT. The documentation says:
Maybe I'm reading too much into this, but if I switch from presenting framebuffer A to presenting framebuffer B by issuing a non-blocking atomic commit and I wait for the out fence file descriptor returned by the affected CRTC, I expect fb B to be on screen while I can safely drmModeRmFB A, as it should no longer be on the screen? Is that not correct? The sequence of events is:This Sync File contains the CRTC fence that will be signaled when all framebuffers present on the Atomic Commit * request for that given CRTC are scanned out on the screen.
- Create a new atomic commit containing:
- Replace FB_ID on a plane from FB A with FB B. Save reference to FB A for later
- Add OUT_FENCE_PTR for the CRTC of FB B's plane.
- Do atomic non-blocking commit
- Save the fence file descriptor now filled in for later
- Prepare everything for the next frame (which might take a few ms)
- poll on the fence fd, which, depending on the time taken by the previous steps takes from 1 to 16ms.
- Now drmModeRmFB frame buffer A. The kernel now complains.
messages in the kernel log instead. Though only for video frame buffers, not GL frame buffers for some reason (EDIT: Ah. I recycle the GL framebuffer by handing it back to mesa using gbm_surface_release_buffer, so it's not actually drmModeRmFB'ed). It seems waiting for the fence isn't enough. Am I misunderstanding the documentation?vc4-drm axi:gpu: [drm:drm_mode_rmfb_work_fn [drm]] Removing [FB:688] from all active usage due to RMFB ioctl
info-beamer hosted - A user and programmer friendly digital signage platform for the Pi: https://info-beamer.com/hosted
- 6by9
- Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator - Posts: 18477
- Joined: Wed Dec 04, 2013 11:27 am
Re: Seeking insight about the framebuffer lifecycle
Does GL actually call drmModeRmFB on buffers in normal use? It's more normal to map eg 3 buffers and just swap between them. Likewise for video decode but with more buffers.
Calling RMFB doesn't free the underlying allocation, it only removes the DRM handle for the buffer.
I know very little about that fence handling as it's done by the core code. I'd normally associate it with a writeback connector rather than a live output, but it may work with both.
I suspect that you're after a dma_fence_signal() call which will actually signal your application. If you're doing the RMFB in that callback context, then I wouldn't want to guarantee that the state has been fully processed and therefore refcounts have hit 0 before the callback completes.
I think you're using the path
drm_crtc_send_vblank_event calls send_vblank_event,
which calls drm_send_event_timestamp_locked,
which calls drm_send_event_helper,
and that calls dma_fence_signal or dma_fence_signal_timestamp.
Add some logging in the kernel to show when the fence is signaled vs when you call RMFB, and log where RMFB thinks the buffer is still used.
Calling RMFB doesn't free the underlying allocation, it only removes the DRM handle for the buffer.
I know very little about that fence handling as it's done by the core code. I'd normally associate it with a writeback connector rather than a live output, but it may work with both.
I suspect that you're after a dma_fence_signal() call which will actually signal your application. If you're doing the RMFB in that callback context, then I wouldn't want to guarantee that the state has been fully processed and therefore refcounts have hit 0 before the callback completes.
I think you're using the path
drm_crtc_send_vblank_event calls send_vblank_event,
which calls drm_send_event_timestamp_locked,
which calls drm_send_event_helper,
and that calls dma_fence_signal or dma_fence_signal_timestamp.
Add some logging in the kernel to show when the fence is signaled vs when you call RMFB, and log where RMFB thinks the buffer is still used.
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
Re: Seeking insight about the framebuffer lifecycle
Thanks for your response and the linked code path :)
It seems both the DRM_MODE_PAGE_FLIP_EVENT and the fencing signal complete while the commit is between these two steps:
Now it's essentially a race. If I call rmfb fast enough before the atomic commit state is free'd, I get that extra "Removing [FB:xxx] from all active usage due to RMFB ioctl" message. If the call happens to trigger after the atomic state is free'd, the message doesn't happen. So usleep(100) greatly reduces the number of times the warning appears. :-\. I'm not sure there is any other trigger I might use to free buffers. In my code, once I've waited for the previous commit to complete, I immediately assemble the next commit and commit it. The only other point might be to defer the rmfb calls until the next commit was submitted. I'll try that.
Nope. You're correct. I edited my post, probably while you wrote your response. That does at least explain why I'm not seeing these messages for GL framebuffers as they get indeed recycled.Does GL actually call drmModeRmFB on buffers in normal use?
It seems both the DRM_MODE_PAGE_FLIP_EVENT and the fencing signal complete while the commit is between these two steps:
Code: Select all
[ 5154.206904] vc4-drm axi:gpu: [drm:drm_atomic_state_default_clear [drm]] Clearing atomic state 0000000026205e92
my code gets signaled here...
[ 5154.207217] vc4-drm axi:gpu: [drm:__drm_atomic_state_free [drm]] Freeing atomic state 0000000026205e92
info-beamer hosted - A user and programmer friendly digital signage platform for the Pi: https://info-beamer.com/hosted
Re: Seeking insight about the framebuffer lifecycle
That worked.
The GL API makes this trivial, but I'm currently only reusing dma bufs (in case of software decoding H264 on Pi5), but don't reuse framebuffers at the moment. I would have to check how to properly keep references after handing back avframes to the decoder. That's for another day maybe :)It's more normal to map eg 3 buffers and just swap between them. Likewise for video decode but with more buffers.
info-beamer hosted - A user and programmer friendly digital signage platform for the Pi: https://info-beamer.com/hosted
- 6by9
- Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator - Posts: 18477
- Joined: Wed Dec 04, 2013 11:27 am
Re: Seeking insight about the framebuffer lifecycle
I normally defer to JohnC on that one, and it'll depend on the decode framework you're using. Mention of avframes implies libavcodec.dividuum wrote: ↑Thu Feb 06, 2025 3:36 pmThat worked.
The GL API makes this trivial, but I'm currently only reusing dma bufs (in case of software decoding H264 on Pi5), but don't reuse framebuffers at the moment. I would have to check how to properly keep references after handing back avframes to the decoder. That's for another day maybe :)It's more normal to map eg 3 buffers and just swap between them. Likewise for video decode but with more buffers.
https://github.com/jc-kynesim/hello_drmprime/ was his demo using libavcodec and DRM. He has a drm helper library in drmu, which I'm sure had a hello_drmu example, however I can't see it immediately now.
The other one to look at is Kodi as it's obviously doing the same thing.
https://github.com/xbmc/xbmc/blob/maste ... MUtils.cpp has the helper functions for AddFB and RmFB, but I've not looked into how they tag buffers.
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
Re: Seeking insight about the framebuffer lifecycle
Thanks for the links. I took a peek at drmu, like I did occasionally in the past, but from what I understand it doesn't keep framebuffers around in case of hardware decoding. Looking at drmprime_video_display, the code differs base on whether it's a hardware frame or a software decoded frame. While the latter uses a custom get_buffer2 handler to create a pool of framebuffers, the former seems to use drmu_fb_av_new_frame_attach to attach a new framebuffer to each AVFrame. As that happens unconditionally for every frame presented, I guess it's unref'ed somewhere before it's being returned to the decoder. Not 100% sure about that though.
I could probably do something similar for the software decoded path, like adding framebuffers to the already pooled dma bufs I'm using in that case. Still not sure how to do that for hardware decoded AVFrames. Not sure if it would be worth it as the few syscalls needed (3xDRM_IOCTL_PRIME_FD_TO_HANDLE + 1xDRM_IOCTL_MODE_ADDFB2) are rather fast and right now the code path is nicely unified regardless of type of frame (HW/SW) *and* way of presenting them (GL or DRM). I guess I'll keep that for now :}
I could probably do something similar for the software decoded path, like adding framebuffers to the already pooled dma bufs I'm using in that case. Still not sure how to do that for hardware decoded AVFrames. Not sure if it would be worth it as the few syscalls needed (3xDRM_IOCTL_PRIME_FD_TO_HANDLE + 1xDRM_IOCTL_MODE_ADDFB2) are rather fast and right now the code path is nicely unified regardless of type of frame (HW/SW) *and* way of presenting them (GL or DRM). I guess I'll keep that for now :}
info-beamer hosted - A user and programmer friendly digital signage platform for the Pi: https://info-beamer.com/hosted
- 6by9
- Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator - Posts: 18477
- Joined: Wed Dec 04, 2013 11:27 am
Re: Seeking insight about the framebuffer lifecycle
If you're happy to just defer RmFB for an extra frame, then that's the easiest solution.
I don't know the full details of what you get out of avcodec in terms of handle.
V4L2 obviously has the buffer index and recycles the same buffers.
You could go for a buffer mapping cache so that it just retains the mappings until some flushing operation (stop?) comes along and releases them all, but that does have the potential to end up hogging a lot of memory if it goes wrong.
I don't know the full details of what you get out of avcodec in terms of handle.
V4L2 obviously has the buffer index and recycles the same buffers.
You could go for a buffer mapping cache so that it just retains the mappings until some flushing operation (stop?) comes along and releases them all, but that does have the potential to end up hogging a lot of memory if it goes wrong.
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
Return to "Graphics programming"
Jump to
- Community
- General discussion
- Announcements
- Other languages
- Deutsch
- Español
- Français
- Italiano
- Nederlands
- 日本語
- Polski
- Português
- Русский
- Türkçe
- User groups and events
- Raspberry Pi Official Magazine
- Using the Raspberry Pi
- Beginners
- Troubleshooting
- Advanced users
- Assistive technology and accessibility
- Education
- Picademy
- Teaching and learning resources
- Staffroom, classroom and projects
- Astro Pi
- Mathematica
- High Altitude Balloon
- Weather station
- Programming
- C/C++
- Java
- Python
- Scratch
- Other programming languages
- Windows 10 for IoT
- Wolfram Language
- Bare metal, Assembly language
- Graphics programming
- OpenGLES
- OpenVG
- OpenMAX
- General programming discussion
- Projects
- Networking and servers
- Automation, sensing and robotics
- Graphics, sound and multimedia
- Other projects
- Media centres
- Gaming
- AIY Projects
- Hardware and peripherals
- Camera board
- Compute Module
- Official Display
- HATs and other add-ons
- Device Tree
- Interfacing (DSI, CSI, I2C, etc.)
- Keyboard computers (400, 500, 500+)
- Raspberry Pi Pico
- General
- SDK
- MicroPython
- Other RP2040 boards
- Zephyr
- Rust
- AI Accelerator
- AI Camera - IMX500
- Hailo
- Software
- Raspberry Pi OS
- Raspberry Pi Connect
- Raspberry Pi Desktop for PC and Mac
- Beta testing
- Other
- Android
- Debian
- FreeBSD
- Gentoo
- Linux Kernel
- NetBSD
- openSUSE
- Plan 9
- Puppy
- Arch
- Pidora / Fedora
- RISCOS
- Ubuntu
- Ye Olde Pi Shoppe
- For sale
- Wanted
- Off topic
- Off topic discussion