Add Audio Support #3

divyaranjan commented 2026年01月29日 23:13:11 +01:00

Right now when PALE displays a video, it just does that. There are two ways to go about adding audio:

1. "Cheap Way": Just add a few more flags to ffmpeg to make it stream the sound to the default audio output of the system and call it a day. I have tried this, and one can try this with the following patch on GNU/Linux systems where Pulseaudio (through Pipewire or otherwise) is used:

diff --git a/pale-video.c b/pale-video.c
index f4743d1..b4004d7 100644
--- a/pale-video.c
+++ b/pale-video.c
@@ -296,10 +296,16 @@ pale_video_create(const char *video_path, PaleImage *target_image)
 		snprintf(size_str, sizeof(size_str), "%dx%d", player->width,
 			 player->height);
 
-		execlp("ffmpeg", "ffmpeg", "-i", video_path, "-vf", "vflip",
-		 "-f", "rawvideo", "-pix_fmt", "rgb24", "-s",
-		 size_str, /* Scale to target size */
-		 "-", NULL);
+		execlp("ffmpeg", "ffmpeg",
+		 "-re", /* Read at native frame rate for sync */
+		 "-i", video_path,
+
+		 /* video to pipe */
+		 "-map", "0:v", "-vf", "vflip", "-f", "rawvideo",
+		 "-pix_fmt", "rgb24", "-s", size_str, "-",
+
+		 /* audio to system speakers */
+		 "-map", "0:a", "-f", "pulse", "default", NULL);
 
 		/* If exec fails */
 		_exit(1);

You can replace the pulse with alsa if you don't use the former. The issue with this is that the audio is noticeably out of sync with the frames of the video. I have attached a demo that reproduces this. Also to be noted that this seems cheap way also seems to introduce slight lags in the video frames as well.

2. Proper AV Sync: This is the actually efficient way to implement this. It would entail tracking a Master Clock, usually the audio clock is considered the Master and then we try to make sure the video frames are aligned according to it. Also we'd need to output the audio from ffmpeg in some interleaved(?) format to receive it so we'll need mutlipe pipes that do both audio and video. And then we need to process the audio samples through another dynamically linked audio library and then do the syncing.

Obviously we need to do the second approach, but the choice of audio library, the clock tracking architecture and other details need to be discussed.

Right now when PALE displays a video, it just does that. There are two ways to go about adding audio: 1. "Cheap Way": Just add a few more flags to `ffmpeg` to make it stream the sound to the default audio output of the system and call it a day. I have tried this, and one can try this with the following patch on GNU/Linux systems where Pulseaudio (through Pipewire or otherwise) is used: ``` diff --git a/pale-video.c b/pale-video.c index f4743d1..b4004d7 100644 --- a/pale-video.c +++ b/pale-video.c @@ -296,10 +296,16 @@ pale_video_create(const char *video_path, PaleImage *target_image) snprintf(size_str, sizeof(size_str), "%dx%d", player->width, player->height); - execlp("ffmpeg", "ffmpeg", "-i", video_path, "-vf", "vflip", - "-f", "rawvideo", "-pix_fmt", "rgb24", "-s", - size_str, /* Scale to target size */ - "-", NULL); + execlp("ffmpeg", "ffmpeg", + "-re", /* Read at native frame rate for sync */ + "-i", video_path, + + /* video to pipe */ + "-map", "0:v", "-vf", "vflip", "-f", "rawvideo", + "-pix_fmt", "rgb24", "-s", size_str, "-", + + /* audio to system speakers */ + "-map", "0:a", "-f", "pulse", "default", NULL); /* If exec fails */ _exit(1); ``` You can replace the `pulse` with `alsa` if you don't use the former. The issue with this is that the audio is noticeably out of sync with the frames of the video. I have attached a demo that reproduces this. Also to be noted that this seems cheap way also seems to introduce slight lags in the video frames as well. 2. Proper AV Sync: This is the actually efficient way to implement this. It would entail tracking a Master Clock, usually the audio clock is considered the Master and then we try to make sure the video frames are aligned according to it. Also we'd need to output the audio from `ffmpeg` in some interleaved(?) format to receive it so we'll need mutlipe pipes that do both `audio` and `video`. And then we need to process the audio samples through another dynamically linked audio library and then do the syncing. Obviously we need to do the second approach, but the choice of audio library, the clock tracking architecture and other details need to be discussed.

appetrosyan commented 2026年01月30日 01:19:41 +01:00

This may become challenging. What you probably want to do is expose some knobs so you can properly use the sub-process to stream the audio as it is done in e.g. Chrome, or Firefox.

This would make the Elisp boiler-plate heavy, but that can be hidden in Elisp very easily to get default behaviour.

This may become challenging. What you probably want to do is expose some knobs so you can properly use the sub-process to stream the audio as it is done in e.g. Chrome, or Firefox. This would make the Elisp boiler-plate heavy, but that can be hidden in Elisp very easily to get default behaviour.

appetrosyan commented 2026年01月30日 01:22:44 +01:00

Another issue is that mpv actually does quite a lot of clever things that are not easy to do even if we get the library upstreamed. Specifically, there's this thing called display-resample which basically means that it can adaptively stretch and squeeze the audio to match the display refresh rate and render pace. This does wonders on complex codecs like H266 or AV1 that have non-trivial decoding and can have frame-pacing issues.

We need this doubly, because we have the GC that can freeze the thread with the canvas. Unless we make the canvas completely asynchronous from the Elisp thread, which means implementing a proper event loop.... which is kinda the point of NeoEmacs

Another issue is that `mpv` actually does quite a lot of clever things that are not easy to do even if we get the library upstreamed. Specifically, there's this thing called `display-resample` which basically means that it can adaptively stretch and squeeze the audio to match the display refresh rate and render pace. This does wonders on complex codecs like H266 or AV1 that have non-trivial decoding and can have frame-pacing issues. We need this doubly, because we have the GC that can freeze the thread with the canvas. Unless we make the canvas completely asynchronous from the Elisp thread, which means implementing a proper event loop.... which is kinda the point of NeoEmacs

divyaranjan commented 2026年01月31日 05:38:45 +01:00

@appetrosyan wrote in #3 (comment):

This may become challenging. What you probably want to do is expose some knobs so you can properly use the sub-process to stream the audio as it is done in e.g. Chrome, or Firefox.

Yeah, we can already use the ffmpeg to stream the audio, the issue is the syncing.

@appetrosyan wrote in https://codeberg.org/MonadicSheep/pale/issues/3#issuecomment-10255980: > This may become challenging. What you probably want to do is expose some knobs so you can properly use the sub-process to stream the audio as it is done in e.g. Chrome, or Firefox. Yeah, we can already use the `ffmpeg` to stream the audio, the issue is the syncing.

divyaranjan commented 2026年01月31日 05:57:51 +01:00

Some resources I found, might be worth looking into:

1. Ryan Gordon's Building an SDL Media Player Playlist
2. https://github.com/DeshyDan/Not-VLC
3. https://github.com/rambod-rahmani/ffmpeg-video-player (https://dranger.com/ffmpeg/)

Some resources I found, might be worth looking into: 1. Ryan Gordon's [Building an SDL Media Player Playlist](https://www.youtube.com/playlist?list=PL6m6sxLnXksbqdsAcpTh4znV9j70WkmqG) 2. https://github.com/DeshyDan/Not-VLC 3. https://github.com/rambod-rahmani/ffmpeg-video-player (https://dranger.com/ffmpeg/)

appetrosyan commented 2026年03月18日 11:58:45 +01:00

Some findings from neoemacs that might be relevant here.

MPV doesn't sync by adjusting the frame timing to the display's refresh rate (proportionately, not exactly), and does this by resampling the audio.

This is something that we should do, but something that would require upstream changes to the canvas API. While work on that is on-going, it'd be wise to consider this

Some findings from neoemacs that might be relevant here. MPV doesn't sync by adjusting the frame timing to the display's refresh rate (proportionately, not exactly), and does this by resampling the audio. This is something that we should do, but something that would require upstream changes to the canvas API. While work on that is on-going, it'd be wise to consider this