Skip to content

Navigation Menu

Sign in
Sign up

Add video-file playback to HTMLVideoElement (src) for VideoTexture - #1770

Draft
julapy wants to merge 2 commits into
BabylonJS:master from
julapy:video-file-playback
Draft

Add video-file playback to HTMLVideoElement (src) for VideoTexture #1770
julapy wants to merge 2 commits into
BabylonJS:master from
julapy:video-file-playback

Conversation

@julapy

@julapy julapy commented Jul 7, 2026

Copy link
×ばつ2000 h264 clip on two independent `HTMLVideoElement`s, both looping, one seeked out of phase mid-run to exercise the control API — sustained 60 FPS (~1 ms GPU). Also runs concurrently with an active `immersive-ar` session and WebXR raw-camera-access plane in the same scene. 🤖 Generated with [Claude Code](https://claude.com/claude-code) " data-view-component="true"> Copy Markdown

What

Adds video-file playback to HTMLVideoElement, so Babylon.js content can play a video file/URL and use its frames as a VideoTexture — the same web API, unmodified Babylon.js:

const video = new HTMLVideoElement();
video.src = "app:///assets/clip.mp4"; // or an absolute path / http(s) URL
video.loop = true;
video.muted = true;
const texture = new BABYLON.VideoTexture("vid", video, scene, false, true, undefined, { autoPlay: true, loop: true });
material.emissiveTexture = texture;
// ...standard control API: video.play(), pause(), currentTime = 3.0, video.duration, events, etc.

Previously the HTMLVideoElement polyfill only supported srcObject (a camera MediaStream). This adds the src path alongside it; the two sources are mutually exclusive, matching the web. Any number of videos can play simultaneously (each element owns its own decoder + texture).

How

Commit 1 — VideoPlayer backend:

  • VideoPlayer.h: a small platform abstraction (play/pause/loop/muted/volume/seek, duration/currentTime/dimensions, per-frame UpdateTexture, and load/ended/seeked events).
  • Apple/VideoPlayer.mm: AVPlayer + AVPlayerItemVideoOutput requesting BGRA frames (no YUV pass), viewed zero-copy via CVMetalTextureCache and blitted into a persistent MTLTexture, wired to Babylon's InternalTexture with the same bgfx::overrideInternal-on-render-thread retry pattern as CameraDevice::UpdateCameraTexture. Loops via seek-to-zero.

Commit 2 — HTMLVideoElement.src:

  • Extends NativeVideo with src, currentTime (get/seek), duration, loop, muted, paused, ended, isNative, and events (loadedmetadata/loadeddata/canplay/playing/pause/seeked/ended/resize). Player callbacks (which may fire on any thread) are marshalled onto the JS thread via JsRuntime::Dispatch; the element self-references while it owns a source so it isn't collected mid-playback. UpdateTexture routes to the camera stream or the video player depending on the active source.

Only the Apple backend is implemented here; other platforms return nullptr from VideoPlayer::Create (video src is a no-op, srcObject/camera unaffected).

Testing

Verified on device (iPhone 15 Pro Max, iOS 26.5, Xcode 26, Babylon.js 9.9.1 UMD) via the iOS Playground: a ×ばつ2000 h264 clip on two independent HTMLVideoElements, both looping, one seeked out of phase mid-run to exercise the control API — sustained 60 FPS (~1 ms GPU). Also runs concurrently with an active immersive-ar session and WebXR raw-camera-access plane in the same scene.

🤖 Generated with Claude Code

matthargett reacted with eyes emoji
julapy and others added 2 commits July 7, 2026 18:32 
Adds a platform VideoPlayer abstraction (VideoPlayer.h) plus an Apple
implementation backed by AVPlayer + AVPlayerItemVideoOutput. Frames are
requested as BGRA (no YUV conversion needed), viewed zero-copy through a
CVMetalTextureCache and blitted into a persistent MTLTexture, which is wired
to Babylon's InternalTexture with the same bgfx::overrideInternal-on-render-
thread retry pattern as CameraDevice::UpdateCameraTexture.
Each player instance owns its own decoder and texture, so any number of videos
can play simultaneously. URLs may be app:///<bundle-relative>, absolute file
paths, or http(s). Playback control (play/pause/loop/muted/volume/seek) and
metadata (duration/currentTime/dimensions) are exposed for the JS layer;
end-of-stream loops via seek-to-zero or reports Ended. Event callbacks may fire
on any thread and are documented as caller-marshalled.
This is the decode/GPU half of HTMLVideoElement.src support.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
...API)
Extends the HTMLVideoElement polyfill so a video file/URL can be played and
its frames rendered into the Babylon scene, alongside the existing srcObject
(camera MediaStream) source. Assigning src creates a VideoPlayer; the two
sources are mutually exclusive, matching the web API.
Implements the HTMLVideoElement surface Babylon.js's VideoTexture and typical
app code use: src, currentTime (get/seek), duration, loop, muted, paused,
ended, videoWidth/videoHeight, plus events (loadedmetadata, loadeddata,
canplay, playing, pause, seeked, ended, resize). isNative is reported so
VideoTexture accepts the element directly. VideoPlayer event callbacks are
marshalled onto the JS thread via JsRuntime::Dispatch; the element holds a
self-reference while it owns a source so it stays alive during playback.
UpdateTexture pulls decoded frames each render frame and raises resize on
dimension changes.
Babylon.js is unmodified: new BABYLON.VideoTexture(name, videoElement, scene)
with videoElement.src set works as on the web, including multiple simultaneous
videos. Verified on device (iPhone 15 Pro Max, iOS 26.5) with two independent
looping videos plus an out-of-phase seek at 60 FPS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copy link
Copy Markdown
Member

Latest bgfx can do hardware video decoding on D3D11, D3D12, Vulkan, and Metal. You might want to check that feature so that feature it's not macOS only.

julapy commented Jul 7, 2026

Copy link
Copy Markdown
Author

hi @bkaradzic-microsoft - "latest bgfx" - is that already in BN and is it just a matter of hooking up video decoding?

ryantrem commented Jul 8, 2026

Copy link
Copy Markdown
Member

Yes, bgfx is the rendering abstraction (over D3D, OpenGL, Vulkan, Metal) that Babylon Native uses.

Copy link
Copy Markdown
Member

@julapy I need sync with latest and then HW video decoding will be available in BN.

Copy link
Copy Markdown
Member

@julapy I updated CMake to get latest bgfx which has HW video decoder support. You would need to enable BGFX_CONFIG_VIDEO:

target_compile_definitions(bgfx PRIVATE BGFX_CONFIG_VIDEO=0)

And there is example of video player:
https://github.com/BabylonJS/bgfx/blob/539e1febfba9c0ad85cee43266be9bb066225b3e/tools/texturev/video_player.cpp

julapy commented Jul 10, 2026

Copy link
Copy Markdown
Author

@bkaradzic-microsoft thanks for syncing bgfx and pointing me at the gate + the texturev player — I went through video.h, video_mtl.cpp and the example to understand the shape of it.

If I've read it right: bgfx::VideoDecoder takes parameter sets + elementary-stream access units with PTS and hardware-decodes into a bgfx texture (VideoToolbox on Metal, D3D/Vulkan video elsewhere), and everything above the bitstream — container demux (l-smash in the example), pacing/clock, seek/loop — stays app-side. That's a really clean split, and it maps nicely onto this PR's structure: VideoPlayer.h is deliberately a small platform abstraction and the Apple AVPlayer file is just one implementation of it. A bgfx-video-based VideoPlayer looks like the natural way to fill in the platforms that currently return nullptr from VideoPlayer::Create — one implementation for D3D11/D3D12/Vulkan instead of per-platform media APIs, and it would land frames in a bgfx texture directly instead of the overrideInternal dance this PR borrows from the camera path.

Two questions before I go there, since HTMLVideoElement parity is the target:

  1. Audio — I couldn't find an audio path in the decoder or the texturev player. Is audio intentionally out of scope for bgfx::VideoDecoder (i.e. apps bring their own audio decode + A/V sync clock), or is there something planned? Most content we render has audio baked into the video, which is a big part of why the Apple implementation sits on AVPlayer (demux, buffering, audio and sync come for free).

  2. Network sources — the example reads local files through l-smash. For video.src = "https://..." semantics we'd need progressive download/buffered reads; is a custom l-smash reader the intended route, or would you expect apps to download-then-play?

Depending on the answers, my thinking would be: keep AVPlayer as the Apple backend (audio + streaming for free, zero added binary), and add a bgfx-video VideoPlayer for the other platforms in a follow-up PR — happy to take that on. Also glad to flip BGFX_CONFIG_VIDEO on in that follow-up rather than this one so this PR stays scoped.

Copy link
Copy Markdown
Member

@julapy

  1. Yes, there is no audio part because bgfx doesn't deal with sound, this is just HW video decoding. Audio should be routed via something like Miniaudio: Add partial support to Audio through Miniaudio #1765

  2. No, it's not expected to download then play, rather video player should be modified to allow buffered reads.

Depending on the answers, my thinking would be: keep AVPlayer as the Apple backend (audio + streaming for free, zero added binary), and add a bgfx-video VideoPlayer for the other platforms in a follow-up PR — happy to take that on. Also glad to flip BGFX_CONFIG_VIDEO on in that follow-up rather than this one so this PR stays scoped.

We don't want Apple only solution. We need something that would work across all other platforms (at least one more OS, for example Windows should be supported in this PR). It might be larger work to get you where you need to be but in order to accept this change, we need cross-platform solution.

bghgary marked this pull request as draft July 22, 2026 16:29

bghgary commented Jul 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Converting to draft per Branimir's feedback. Mark ready again once that's in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /