WIP: Integrate Canvas API #10

divyaranjan commented 2026年02月02日 23:03:46 +01:00

This is going to be a long PR which would try to integrate the Canvas API that was introduced by Daniel:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=80281

You should test this branch by building Emacs from the following fork which has the above patches applied and updated:

https://codeberg.org/MonadicSheep/emacs

This PR would remove the following from Pale which would be redundant after using the Canvas API:

• Tiling
• Overlay hash maps
• Converting textures to PPMs

Closes #9

This is going to be a long PR which would try to integrate the Canvas API that was introduced by Daniel: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=80281 You should test this branch by building Emacs from the following fork which has the above patches applied and updated: https://codeberg.org/MonadicSheep/emacs This PR would remove the following from Pale which would be redundant after using the Canvas API: - Tiling - Overlay hash maps - Converting textures to PPMs Closes #9

tusharhero commented 2026年02月03日 12:20:03 +01:00

If we are going to support both Emacs 30 and the future version of Emacs with the canvas API, we would need some sort of mechanism to automatically detect that. And since the header files are different, we should probably have two emacs-module.h files, one for each version.

WDYT @appetrosyan @appetrosyan ?

If we are going to support both Emacs 30 and the future version of Emacs with the canvas API, we would need some sort of mechanism to automatically detect that. And since the header files are different, we should probably have two emacs-module.h files, one for each version. WDYT @appetrosyan @appetrosyan ?

minad commented 2026年02月03日 17:45:55 +01:00

I think one can use the emacs-module.h from Emacs 31 with an adjusted size check. If Emacs 31 is not available, just don't use the new canvas function. My suggestion would be to support a simplistic fallback with a single PPM on old Emacs versions and avoid the complexity with the tiling. It is okay if a new package requires new Emacs features for best performance. Emacs is actually moving quite fast these days - I'd expect that the emacs-31 branch will be cut in about half a year or year.

I think one can use the emacs-module.h from Emacs 31 with an adjusted size check. If Emacs 31 is not available, just don't use the new canvas function. My suggestion would be to support a simplistic fallback with a single PPM on old Emacs versions and avoid the complexity with the tiling. It is okay if a new package requires new Emacs features for best performance. Emacs is actually moving quite fast these days - I'd expect that the emacs-31 branch will be cut in about half a year or year.

tusharhero commented 2026年02月04日 10:08:52 +01:00

Hello @minad, if the idea is to make emacs-reader use this library, falling back to a single PPM is simply not possible. The text-selection features will start consuming a humongous amount of memory eventually ending in an oom kill. The text-features branch of emacs-reader is currently using m-mapped file for quickly switching the images for highlighting (and on top of that it retains the old: convert ppm string to image approach for other things), which is quite ugly but we had no choice. The plan with that is to release the basic text-features with this approach for 0.4.0 sometime this month, and then start working to integrate emacs-reader with pale.

Hello @minad, if the idea is to make emacs-reader use this library, falling back to a single PPM is simply not possible. The text-selection features will start consuming a humongous amount of memory eventually ending in an oom kill. The `text-features` branch of emacs-reader is currently using m-mapped file for quickly switching the images for highlighting (and on top of that it retains the old: convert ppm string to image approach for other things), which is quite ugly but we had no choice. The plan with that is to release the basic text-features with this approach for 0.4.0 sometime this month, and then start working to integrate emacs-reader with pale.

tusharhero commented 2026年02月04日 10:18:12 +01:00

@tusharhero wrote in #10 (comment):

using m-mapped file for quickly switching the images

@divyaranjan, how do you feel about using this approach for PALE's fallback? Tiling will cause us many pains because it is complicated (also on the front-end side.)

CC: @appetrosyan

@tusharhero wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10356710: > using m-mapped file for quickly switching the images @divyaranjan, how do you feel about using this approach for PALE's fallback? Tiling will cause us many pains because it is complicated (also on the front-end side.) CC: @appetrosyan

minad commented 2026年02月04日 12:05:27 +01:00

@tusharhero I think I would be mainly interested in avoiding the tiling. If the canvas and memory-mapped approach both work well enough, I would go that way, since then the code would overall be similar for both APIs, and you would not have to carry around so much legacy code. Another option would be to simply only support Emacs 31 and newer, but it is a tough sell I guess. Do you have some numbers? How does mmapped to the canvas approach compare?

@tusharhero I think I would be mainly interested in avoiding the tiling. If the canvas and memory-mapped approach both work well enough, I would go that way, since then the code would overall be similar for both APIs, and you would not have to carry around so much legacy code. Another option would be to simply only support Emacs 31 and newer, but it is a tough sell I guess. Do you have some numbers? How does mmapped to the canvas approach compare?

divyaranjan commented 2026年02月04日 12:28:36 +01:00

@minad wrote in #10 (comment):

@tusharhero I think I would be mainly interested in avoiding the tiling. If the canvas and memory-mapped approach both work well enough, I would go that way, since then the code would overall be similar for both APIs, and you would not have to carry around so much legacy code. Another option would be to simply only support Emacs 31 and newer, but it is a tough sell I guess. Do you have some numbers? How does mmapped to the canvas approach compare?

Tushar is talking about mmap and tiling as options when you don't have the Canvas API, that is as a fallback for older Emacs versions. All the hacks we've done, mmaping tmpfiles or tiling are inferior to the Canvas API.

@minad wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10359958: > @tusharhero I think I would be mainly interested in avoiding the tiling. If the canvas and memory-mapped approach both work well enough, I would go that way, since then the code would overall be similar for both APIs, and you would not have to carry around so much legacy code. Another option would be to simply only support Emacs 31 and newer, but it is a tough sell I guess. Do you have some numbers? How does mmapped to the canvas approach compare? Tushar is talking about mmap and tiling as options when you don't have the Canvas API, that is as a fallback for older Emacs versions. All the hacks we've done, mmaping tmpfiles or tiling are inferior to the Canvas API.

divyaranjan commented 2026年02月04日 12:38:58 +01:00

I think before merging this PR into Pale, we should relese a version of master and make Reader use that version of Pale whenever it is used in Emacs versions less than 31.

So it will do exactly what master does currently, which is GPU + PPM tiling. All we'd need to do is add some more functions to make sure it works with Reader. Essentially we'd have to decide that Pale for older Emacs versions would be limited both in performance and API.

I think before merging this PR into Pale, we should relese a version of master and make Reader use that version of Pale whenever it is used in Emacs versions less than 31. So it will do exactly what `master` does currently, which is GPU + PPM tiling. All we'd need to do is add some more functions to make sure it works with Reader. Essentially we'd have to decide that Pale for older Emacs versions would be limited both in performance and API.

minad commented 2026年02月04日 12:41:59 +01:00

Tushar is talking about mmap and tiling as options when you don't have the Canvas API, that is as a fallback for older Emacs versions.

Yes, of course. I understood that.

All the hacks we've done, mmaping tmpfiles or tiling are inferior to the Canvas API.

My motivation is to keep the amount of such hacks as low as possible, and in particular not let the design be guided from such hacks in the light of a cleaner solution in the form of the canvas api. What I am trying to say is that a slightly inferior behavior is acceptable on older Emacs versions, but of course OOM crashes go too far. Then I think only supporting Emacs 31+ would be acceptable as well, but I am not a typical user.

> Tushar is talking about mmap and tiling as options when you don't have the Canvas API, that is as a fallback for older Emacs versions. Yes, of course. I understood that. > All the hacks we've done, mmaping tmpfiles or tiling are inferior to the Canvas API. My motivation is to keep the amount of such hacks as low as possible, and in particular not let the design be guided from such hacks in the light of a cleaner solution in the form of the canvas api. What I am trying to say is that a slightly inferior behavior is acceptable on older Emacs versions, but of course OOM crashes go too far. Then I think only supporting Emacs 31+ would be acceptable as well, but I am not a typical user.

tusharhero commented 2026年02月04日 13:16:56 +01:00

@divyaranjan wrote in #10 (comment):

All the hacks we've done, mmaping tmpfiles or tiling are inferior to the Canvas API.

Yup, but I think m-map is a better compromise.

@minad wrote in #10 (comment):

How does mmapped to the canvas approach compare?

canvas is obviously better, but you can try the emacs-reader text-features branch to get a feel for it. It gets a bit sluggish when you zoom in too much.

@divyaranjan wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10360226: > All the hacks we've done, mmaping tmpfiles or tiling are inferior to the Canvas API. Yup, but I think m-map is a better compromise. @minad wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10359958: > How does mmapped to the canvas approach compare? canvas is obviously better, but you can try the emacs-reader `text-features` branch to get a feel for it. It gets a bit sluggish when you zoom in too much.

minad commented 2026年02月04日 13:36:07 +01:00

Yup, but I think m-map is a better compromise.

Avoiding the tiling seems important to me since for the canvas API we also don't have that.

canvas is obviously better, but you can try the emacs-reader text-features branch to get a feel for it. It gets a bit sluggish when you zoom in too much.

How much better is it? Do you have some hard numbers, also in the light of the discussion on the Emacs bug tracker?

> Yup, but I think m-map is a better compromise. Avoiding the tiling seems important to me since for the canvas API we also don't have that. > canvas is obviously better, but you can try the emacs-reader text-features branch to get a feel for it. It gets a bit sluggish when you zoom in too much. How much better is it? Do you have some hard numbers, also in the light of the discussion on the Emacs bug tracker?

tusharhero commented 2026年02月04日 13:37:52 +01:00

@minad wrote in #10 (comment):

How much better is it? Do you have some hard numbers, also in the light of the discussion on the Emacs bug tracker?

No hard numbers, I guess we will have to add it to pale first, and then run an animation here to test (with FPS).

@minad wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10361040: > > > > How much better is it? Do you have some hard numbers, also in the light of the discussion on the Emacs bug tracker? No hard numbers, I guess we will have to add it to pale first, and then run an animation here to test (with FPS).

minad commented 2026年02月04日 13:56:55 +01:00

We could also write a quick demo based on pale or canvas. For canvas all that is needed is this:

emacs_value Qnil;
int plugin_is_GPL_compatible;
static emacs_value demo_render(emacs_env* env, ptrdiff_t nargs,
 emacs_value args[], void* data) {
 uint32_t* pixel = env->canvas_pixel(env, args[0]);
 // Do some rendering here...
 memset(pixel, 0, 800*600*4);
 return Qnil;
}
int emacs_module_init(struct emacs_runtime *rt) {
 if ((size_t)rt->size < sizeof (*rt))
 return 1;
 emacs_env* env = rt->get_environment(rt);
 if ((size_t)env->size < sizeof (*env))
 return 2;
 Qnil = env->make_global_ref(env, env->intern(env, "nil"));
 env->funcall(env, env->intern(env, "defalias"), 2,
 (emacs_value[]){
 env->intern(env, "demo-render"),
 env->make_function(env, 2, 2, demo_render, 0, 0)
 });
 return 0;
}

(module-load "./demo.so")
(defvar canvas (make-canvas 800 600))
(switch-to-buffer (get-buffer-create "*demo*"))
(setq-local face-remapping-alist '((default (:background "black")))
 cursor-type nil)
(insert (propertize "#" 'display `(image :type canvas :data ,canvas)))
(run-with-timer nil 0.01
 (lambda ()
 (demo-render canvas)
 (canvas-refresh 1)))

We could also write a quick demo based on pale or canvas. For canvas all that is needed is this: ~~~ c emacs_value Qnil; int plugin_is_GPL_compatible; static emacs_value demo_render(emacs_env* env, ptrdiff_t nargs, emacs_value args[], void* data) { uint32_t* pixel = env->canvas_pixel(env, args[0]); // Do some rendering here... memset(pixel, 0, 800*600*4); return Qnil; } int emacs_module_init(struct emacs_runtime *rt) { if ((size_t)rt->size < sizeof (*rt)) return 1; emacs_env* env = rt->get_environment(rt); if ((size_t)env->size < sizeof (*env)) return 2; Qnil = env->make_global_ref(env, env->intern(env, "nil")); env->funcall(env, env->intern(env, "defalias"), 2, (emacs_value[]){ env->intern(env, "demo-render"), env->make_function(env, 2, 2, demo_render, 0, 0) }); return 0; } ~~~ ~~~ emacs-lisp (module-load "./demo.so") (defvar canvas (make-canvas 800 600)) (switch-to-buffer (get-buffer-create "*demo*")) (setq-local face-remapping-alist '((default (:background "black"))) cursor-type nil) (insert (propertize "#" 'display `(image :type canvas :data ,canvas))) (run-with-timer nil 0.01 (lambda () (demo-render canvas) (canvas-refresh 1))) ~~~

minad commented 2026年02月07日 10:31:47 +01:00

Hey @divyaranjan,

I noticed that you start to refactor based on the canvas API. Fyi, I reworked the API, to keep the patch smaller and less intrusive. The main change is that a canvas is represented as an image specification and that redisplay is optimized. This way the canvas is more aligned with the image API and hopefully acceptable upstream. Please check out the newest diff including tests and documentation at https://gist.github.com/minad/486c18c923db7cf1eb0fd5bcc9257796.

The updated example from above looks like this, so no big change:

emacs_value Qnil;
int plugin_is_GPL_compatible;
static emacs_value demo_render(emacs_env* env, ptrdiff_t nargs,
 emacs_value args[], void* data) {
 uint32_t* pixel = env->canvas_pixel(env, args[0]);
 if (pixel) {
 // Do some rendering here...
 memset(pixel, 0, 800*600*4);
 env->canvas_refresh(env, args[0]);
 }
 return Qnil;
}
int emacs_module_init(struct emacs_runtime *rt) {
 if ((size_t)rt->size < sizeof (*rt))
 return 1;
 emacs_env* env = rt->get_environment(rt);
 if ((size_t)env->size < sizeof (*env))
 return 2;
 Qnil = env->make_global_ref(env, env->intern(env, "nil"));
 env->funcall(env, env->intern(env, "defalias"), 2,
 (emacs_value[]){
 env->intern(env, "demo-render"),
 env->make_function(env, 1, 1, demo_render, 0, 0)
 });
 return 0;
}

(module-load "./demo.so")
(switch-to-buffer (get-buffer-create "*demo*"))
(setq canvas `(image :type canvas :canvas-width 800 :canvas-height 600 :canvas-id demo :canvas-buffer ,(current-buffer)))
(insert (propertize "#" 'display canvas))
(run-with-timer nil 0.01 (lambda () (demo-render canvas)))

Regarding the changes here, I have two comments:

1. I suggest to optimize elisp_car by keeping a symbol to Qcar. Interning the string always is quite inefficient.

emacs_value
elisp_car (emacs_env *env, emacs_value list)
{
 return env->funcall (env, Qcar, 1, &list);
}

2. I still wonder about the purpose of the GPU code. You probably want to use it for additional drawing, some transparent overlays, etc? In most cases however if you simply read the video frame or PDF image from ffmpeg or mupdf respectively, push it to the GPU and read it back to the canvas, which is an unnecessary step, or am I missing something?

In case you find time to read through the canvas.diff, I would appreciate feedback. I plan to submit it soon to the Emacs bug tracker again, and probably lots of nit picking will be going on then. So checking it critically helps, with more than my two eyes.

Hey @divyaranjan, I noticed that you start to refactor based on the canvas API. Fyi, I reworked the API, to keep the patch smaller and less intrusive. The main change is that a canvas is represented as an image specification and that redisplay is optimized. This way the canvas is more aligned with the image API and hopefully acceptable upstream. Please check out the newest diff including tests and documentation at https://gist.github.com/minad/486c18c923db7cf1eb0fd5bcc9257796. The updated example from above looks like this, so no big change: ~~~ c emacs_value Qnil; int plugin_is_GPL_compatible; static emacs_value demo_render(emacs_env* env, ptrdiff_t nargs, emacs_value args[], void* data) { uint32_t* pixel = env->canvas_pixel(env, args[0]); if (pixel) { // Do some rendering here... memset(pixel, 0, 800*600*4); env->canvas_refresh(env, args[0]); } return Qnil; } int emacs_module_init(struct emacs_runtime *rt) { if ((size_t)rt->size < sizeof (*rt)) return 1; emacs_env* env = rt->get_environment(rt); if ((size_t)env->size < sizeof (*env)) return 2; Qnil = env->make_global_ref(env, env->intern(env, "nil")); env->funcall(env, env->intern(env, "defalias"), 2, (emacs_value[]){ env->intern(env, "demo-render"), env->make_function(env, 1, 1, demo_render, 0, 0) }); return 0; } ~~~ ~~~ emacs-lisp (module-load "./demo.so") (switch-to-buffer (get-buffer-create "*demo*")) (setq canvas `(image :type canvas :canvas-width 800 :canvas-height 600 :canvas-id demo :canvas-buffer ,(current-buffer))) (insert (propertize "#" 'display canvas)) (run-with-timer nil 0.01 (lambda () (demo-render canvas))) ~~~ Regarding the changes here, I have two comments: 1. I suggest to optimize elisp_car by keeping a symbol to Qcar. Interning the string always is quite inefficient. ~~~ elisp emacs_value elisp_car (emacs_env *env, emacs_value list) { return env->funcall (env, Qcar, 1, &list); } ~~~ 2. I still wonder about the purpose of the GPU code. You probably want to use it for additional drawing, some transparent overlays, etc? In most cases however if you simply read the video frame or PDF image from ffmpeg or mupdf respectively, push it to the GPU and read it back to the canvas, which is an unnecessary step, or am I missing something? In case you find time to read through the canvas.diff, I would appreciate feedback. I plan to submit it soon to the Emacs bug tracker again, and probably lots of nit picking will be going on then. So checking it critically helps, with more than my two eyes.

divyaranjan commented 2026年02月07日 11:40:58 +01:00

@minad I haven't gotten to the part of using Canvas API yet, so I'll update that. But, even if the API uses canvas as an image type, why did you get rid of make-canvas? I think we should have that, or create-canvas.

Philosophically though, a canvas is not an image it's a placeholder for storing image data. I think it should be distinct from the image formats, such as PPM. But this should be discussed in the list.

I still wonder about the purpose of the GPU code. You probably want to use it for additional drawing, some transparent overlays, etc? In most cases however if you simply read the video frame or PDF image from ffmpeg or mupdf respectively, push it to the GPU and read it back to the canvas, which is an unnecessary step, or am I missing something?

Emacs' Canvas API would only provide canvas as an object. You had added canvas-rect too, which I'd like for it to be in core but it could be contentious. Regardless, the goal of Pale is to either wrap whatever the API gives or try to construct them itself.

So, for e.g. if we want to do different forms of blending, something very common as a graphical operation, then we'd need Pale to do it and we'd like it to use the GPU for it if it can. Or, we want Pale to render shaders, or other things where GPU can be helpful.

I agree for simply displaying a framebuffer where we are not computationally manipulating anything, we don't need a GPU, but Pale is supposed to try and support graphical manipulation as a library since applications built on top of Pale might want such things.

For the simple task of displaying a frame of a video, we take ffmpeg's RGB24, we send it to GPU which converts it to ARGB32 and then we point that to the canvas. Similarly, when we'd integrate Pale with reader the simple process of displaying a page would not include any GPU manipulation, it'd simply use the GPU for making sure we get ARGB32 which you would've had to do over CPU anyways.

Do you think this is still unnecessary? On systems without a dedicated or integrated GPU, it'd fallback to software rendering but I still think there's value in providing hardware-accelerated primitives by wrapping around Canvas so that others wouldn't have to reinvent the wheel.

@minad I haven't gotten to the part of using Canvas API yet, so I'll update that. But, even if the API uses canvas as an image type, why did you get rid of `make-canvas`? I think we should have that, or `create-canvas`. Philosophically though, a canvas is not an image it's a placeholder for storing image data. I think it should be distinct from the image formats, such as PPM. But this should be discussed in the list. > I still wonder about the purpose of the GPU code. You probably want to use it for additional drawing, some transparent overlays, etc? In most cases however if you simply read the video frame or PDF image from ffmpeg or mupdf respectively, push it to the GPU and read it back to the canvas, which is an unnecessary step, or am I missing something? Emacs' Canvas API would only provide canvas as an object. You had added `canvas-rect` too, which I'd like for it to be in core but it could be contentious. Regardless, the goal of Pale is to either wrap whatever the API gives or try to construct them itself. So, for e.g. if we want to do different forms of blending, something very common as a graphical operation, then we'd need Pale to do it and we'd like it to use the GPU for it if it can. Or, we want Pale to render shaders, or other things where GPU can be helpful. I agree for simply displaying a framebuffer where we are not computationally manipulating anything, we don't need a GPU, but Pale is supposed to try and support graphical manipulation as a library since applications built on top of Pale might want such things. For the simple task of displaying a frame of a video, we take ffmpeg's RGB24, we send it to GPU which converts it to ARGB32 and then we point that to the canvas. Similarly, when we'd integrate Pale with reader the simple process of displaying a page would not include any GPU manipulation, it'd simply use the GPU for making sure we get ARGB32 which you would've had to do over CPU anyways. Do you think this is still unnecessary? On systems without a dedicated or integrated GPU, it'd fallback to software rendering but I still think there's value in providing hardware-accelerated primitives by wrapping around Canvas so that others wouldn't have to reinvent the wheel.

minad commented 2026年02月07日 11:57:43 +01:00

I haven't gotten to the part of using Canvas API yet, so I'll update that. But, even if the API uses canvas as an image type, why did you get read of make-canvas? I think we should have that, or create-canvas.

We simply don't need make-canvas. The solution works like this - an image specification specifies a canvas with the properties :canvas-id, :canvas-width and :canvas-height. The canvas object is created behind the scene, and if you need to access it, you can simply go through the image specification. This way the API is more aligned with the image/animation API and we don't have to expose additional canvas functions. We could also call it pixbuf or memimage instead of canvas.

You had added canvas-rect too, which I'd like for it to be in core but it could be contentious.

The problem is that it is not clear where to start and where to stop with drawing functions. I don't want to provide plenty of canvas functions since they can be implemented by libraries and we should rather reuse cairo, sdl, skia, agg for this work. Then it is not clear which library to favor. I would rather implement separate canvas-sdl or canvas-cairo modules. The goal of the core functionality is to only provide the necessary glue and nothing else. The simplest solution that works.

For the simple task of displaying a frame of a video, we take ffmpeg's RGB24, we send it to GPU which converts it to ARGB32 and then we point that to the canvas.

Let me be direct - this is not reasonable and will not be efficient. Converting RGB24 to ARGB32 just involves moving bytes around and since you copy the memory anyway, this should be done during copy. I am a bit afraid that things are getting overengineered for no reason. Really, one should use the simplest solution that works.

To be clear, the motivation for my patch was (1) that I had wanted such a canvas for a long time, and (2) that I saw that you went with a route which is too complex (with the tiling, GPU and so on), and I would like that Emacs reader materializes soon on GNU ELPA, so my hope is that I can contribute this way and simplify some things to reduce later headache.

So, for e.g. if we want to do different forms of blending, something very common as a graphical operation, then we'd need Pale to do it and we'd like it to use the GPU for it if it can. Or, we want Pale to render shaders, or other things where GPU can be helpful.

Yes, for this it might be useful, but it is not needed right now, or is it? And simple rectangular overlays can be easily implemented in software, with sufficient performance, as had been shown by my old patch.

I hope this makes sense. I juggled quite some time with the image/canvas/redisplay code to see what works well in Emacs. I can try to explain some things better.

> I haven't gotten to the part of using Canvas API yet, so I'll update that. But, even if the API uses canvas as an image type, why did you get read of make-canvas? I think we should have that, or create-canvas. We simply don't need `make-canvas`. The solution works like this - an image specification specifies a canvas with the properties `:canvas-id`, `:canvas-width` and `:canvas-height`. The canvas object is created behind the scene, and if you need to access it, you can simply go through the image specification. This way the API is more aligned with the image/animation API and we don't have to expose additional canvas functions. We could also call it `pixbuf` or `memimage` instead of `canvas`. > You had added canvas-rect too, which I'd like for it to be in core but it could be contentious. The problem is that it is not clear where to start and where to stop with drawing functions. I don't want to provide plenty of canvas functions since they can be implemented by libraries and we should rather reuse cairo, sdl, skia, agg for this work. Then it is not clear which library to favor. I would rather implement separate canvas-sdl or canvas-cairo modules. The goal of the core functionality is to only provide the necessary glue and nothing else. The simplest solution that works. > For the simple task of displaying a frame of a video, we take ffmpeg's RGB24, we send it to GPU which converts it to ARGB32 and then we point that to the canvas. Let me be direct - this is not reasonable and will not be efficient. Converting RGB24 to ARGB32 just involves moving bytes around and since you copy the memory anyway, this should be done during copy. I am a bit afraid that things are getting overengineered for no reason. Really, one should use the simplest solution that works. To be clear, the motivation for my patch was (1) that I had wanted such a canvas for a long time, and (2) that I saw that you went with a route which is too complex (with the tiling, GPU and so on), and I would like that Emacs reader materializes soon on GNU ELPA, so my hope is that I can contribute this way and simplify some things to reduce later headache. > So, for e.g. if we want to do different forms of blending, something very common as a graphical operation, then we'd need Pale to do it and we'd like it to use the GPU for it if it can. Or, we want Pale to render shaders, or other things where GPU can be helpful. Yes, for this it might be useful, but it is not needed right now, or is it? And simple rectangular overlays can be easily implemented in software, with sufficient performance, as had been shown by my old patch. I hope this makes sense. I juggled quite some time with the image/canvas/redisplay code to see what works well in Emacs. I can try to explain some things better.

divyaranjan commented 2026年02月07日 12:11:04 +01:00

@minad wrote in #10 (comment):

I haven't gotten to the part of using Canvas API yet, so I'll update that. But, even if the API uses canvas as an image type, why did you get read of make-canvas? I think we should have that, or create-canvas.

We simply don't need make-canvas. The solution works like this - an image specification specifies a canvas with the properties :canvas-id, :canvas-width and :canvas-height. The canvas object is created behind the scene, and if you need to access it, you can simply go through the image specification. This way the API is more aligned with the image/animation API and we don't have to expose additional canvas functions. We could also call it pixbuf or memimage instead of canvas.

Understood.

You had added canvas-rect too, which I'd like for it to be in core but it could be contentious.

The problem is that it is not clear where to start and where to stop with drawing functions. I don't want to provide plenty of canvas functions since they can be implemented by libraries and we should rather reuse cairo, sdl, skia, agg for this work. Then it is not clear which library to favor. I would rather implement separate canvas-sdl or canvas-cairo modules. The goal of the core functionality is to only provide the necessary glue and nothing else. The simplest solution that works.

Agreed.

For the simple task of displaying a frame of a video, we take ffmpeg's RGB24, we send it to GPU which converts it to ARGB32 and then we point that to the canvas.

Let me be direct - this is not reasonable and will not be efficient. Converting RGB24 to ARGB32 just involves moving bytes around and since you copy the memory anyway, this should be done during copy. I am a bit afraid that things are getting overengineered for no reason. Really, one should use the simplest solution that works.

Okay, I understood for the conversion going through GPU roundtrip doesn't make sense. But specifically for video, do you think it is not needed at all to provide hardware acceleration, especially when you'd have to render high-res (say above 1920x1080) video? I wanted to keep GPU in the loop for the same reasons video players have hardware acceleration. I think we can probably do that in a better way. Any opinions on how we can do that?

So, for e.g. if we want to do different forms of blending, something very common as a graphical operation, then we'd need Pale to do it and we'd like it to use the GPU for it if it can. Or, we want Pale to render shaders, or other things where GPU can be helpful.

Yes, for this it might be useful, but it is not needed right now, or is it? And simple rectangular overlays can be easily implemented in software, with sufficient performance, as had been shown by my old patch.

It can be sufficient enough, but should we be asking the CPU to do it, in the event that a GPU can do it better? I just don't want to have to go through the trouble in the future of having to re-integrate GPU when we'd need it, so I'd like to have some primitives ready where GPU can be exploited.

We surely don't need it immediately, but I think we soon would. How do you think we should go about it?

@minad wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10425492: > > I haven't gotten to the part of using Canvas API yet, so I'll update that. But, even if the API uses canvas as an image type, why did you get read of make-canvas? I think we should have that, or create-canvas. > > We simply don't need `make-canvas`. The solution works like this - an image specification specifies a canvas with the properties `:canvas-id`, `:canvas-width` and `:canvas-height`. The canvas object is created behind the scene, and if you need to access it, you can simply go through the image specification. This way the API is more aligned with the image/animation API and we don't have to expose additional canvas functions. We could also call it `pixbuf` or `memimage` instead of `canvas`. Understood. > > You had added canvas-rect too, which I'd like for it to be in core but it could be contentious. > > The problem is that it is not clear where to start and where to stop with drawing functions. I don't want to provide plenty of canvas functions since they can be implemented by libraries and we should rather reuse cairo, sdl, skia, agg for this work. Then it is not clear which library to favor. I would rather implement separate canvas-sdl or canvas-cairo modules. The goal of the core functionality is to only provide the necessary glue and nothing else. The simplest solution that works. Agreed. > > For the simple task of displaying a frame of a video, we take ffmpeg's RGB24, we send it to GPU which converts it to ARGB32 and then we point that to the canvas. > > Let me be direct - this is not reasonable and will not be efficient. Converting RGB24 to ARGB32 just involves moving bytes around and since you copy the memory anyway, this should be done during copy. I am a bit afraid that things are getting overengineered for no reason. Really, one should use the simplest solution that works. Okay, I understood for the conversion going through GPU roundtrip doesn't make sense. But specifically for video, do you think it is not needed at all to provide hardware acceleration, especially when you'd have to render high-res (say above 1920x1080) video? I wanted to keep GPU in the loop for the same reasons video players have hardware acceleration. I think we can probably do that in a better way. Any opinions on how we can do that? > > So, for e.g. if we want to do different forms of blending, something very common as a graphical operation, then we'd need Pale to do it and we'd like it to use the GPU for it if it can. Or, we want Pale to render shaders, or other things where GPU can be helpful. > > Yes, for this it might be useful, but it is not needed right now, or is it? And simple rectangular overlays can be easily implemented in software, with sufficient performance, as had been shown by my old patch. It can be sufficient enough, but should we be asking the CPU to do it, in the event that a GPU can do it better? I just don't want to have to go through the trouble in the future of having to re-integrate GPU when we'd need it, so I'd like to have some primitives ready where GPU can be exploited. We surely don't _need_ it immediately, but I think we soon would. How do you think we should go about it?

minad commented 2026年02月07日 12:30:48 +01:00

Okay, I understood for the conversion going through GPU roundtrip doesn't make sense. But specifically for video, do you think it is not needed at all to provide hardware acceleration, especially when you'd have to render high-res (say above 1920x1080) video? I wanted to keep GPU in the loop for the same reasons video players have hardware acceleration. I think we can probably do that in a better way. Any opinions on how we can do that?

I must say I am not an expert on this and I don't know exactly how video players use the gpu. I think some players rely on special hardware decoding, but this might then happen inside ffmpeg. Maybe the video data is already inside the GPU afterwards and can be transferred to display without additional roundtrips.

But if you get the image data out of ffmpeg, you have to pass it again to the GPU to transform it, e.g., distort it, apply shaders, blend, whatever. But this doesn't seem like a common use case, at least not when I watch videos.

Also note that with the Emacs canvas image API we still have to go through memory and the CPU again just to feed the data into Emacs, so this additionally diminishes the advantage of GPU usage.

It can be sufficient enough, but should we be asking the CPU to do it, in the event that a GPU can do it better? I just don't want to have to go through the trouble in the future of having to re-integrate GPU when we'd need it, so I'd like to have some primitives ready where GPU can be exploited.

Yes, I think I prefer the CPU for simple things like blending a few rectangles in a document viewer, since it will just be simpler overall. And I think you won't win anything giving the GPU-CPU roundtrips. It certainly makes sense to measure this, depending on the use case. The situation changes of course for heavier animations, games whatever.

My suggestion would be to consider the use cases separately. Video playing, games and displaying documents are not necessarily the same use case.

In order to get lots of high-qualify drawing operations for free you could simply reuse Cairo here. Cairo image surfaces can use a pixel buffer (canvas_pixel from the Emacs canvas API), and you could then draw directly onto them. But this will usually be in software.

Anyway, I think it makes sense to experiment and try a few different approaches, before settling on a solution. The design space is quite large. From the side of Emacs I am quite happy with the simple API in my newest patch, since most design decisions are pushed outside and libraries/packages have the freedom to implement their own solution as they like. Just exposing the memory pointer is the simplest thing we can do.

In the patch I've added redisplay optimizations, tests and documentation. If you find time to critically read through it, I would really appreciate it, pointing out things you find unclear and so on.

> Okay, I understood for the conversion going through GPU roundtrip doesn't make sense. But specifically for video, do you think it is not needed at all to provide hardware acceleration, especially when you'd have to render high-res (say above 1920x1080) video? I wanted to keep GPU in the loop for the same reasons video players have hardware acceleration. I think we can probably do that in a better way. Any opinions on how we can do that? I must say I am not an expert on this and I don't know exactly how video players use the gpu. I think some players rely on special hardware decoding, but this might then happen inside ffmpeg. Maybe the video data is already inside the GPU afterwards and can be transferred to display without additional roundtrips. But if you get the image data out of ffmpeg, you have to pass it again to the GPU to transform it, e.g., distort it, apply shaders, blend, whatever. But this doesn't seem like a common use case, at least not when I watch videos. Also note that with the Emacs canvas image API we still have to go through memory and the CPU again just to feed the data into Emacs, so this additionally diminishes the advantage of GPU usage. > It can be sufficient enough, but should we be asking the CPU to do it, in the event that a GPU can do it better? I just don't want to have to go through the trouble in the future of having to re-integrate GPU when we'd need it, so I'd like to have some primitives ready where GPU can be exploited. Yes, I think I prefer the CPU for simple things like blending a few rectangles in a document viewer, since it will just be simpler overall. And I think you won't win anything giving the GPU-CPU roundtrips. It certainly makes sense to measure this, depending on the use case. The situation changes of course for heavier animations, games whatever. My suggestion would be to consider the use cases separately. Video playing, games and displaying documents are not necessarily the same use case. In order to get lots of high-qualify drawing operations for free you could simply reuse Cairo here. Cairo image surfaces can use a pixel buffer (canvas_pixel from the Emacs canvas API), and you could then draw directly onto them. But this will usually be in software. Anyway, I think it makes sense to experiment and try a few different approaches, before settling on a solution. The design space is quite large. From the side of Emacs I am quite happy with the simple API in my newest patch, since most design decisions are pushed outside and libraries/packages have the freedom to implement their own solution as they like. Just exposing the memory pointer is the simplest thing we can do. In the patch I've added redisplay optimizations, tests and documentation. If you find time to critically read through it, I would really appreciate it, pointing out things you find unclear and so on.

minad commented 2026年02月07日 13:28:41 +01:00

But I should say that nothing is set in stone, and maybe the canvas API won't make it into Emacs. When I think about the canvas API I see two approaches, the minimal one and a slightly more sophisticated one. I learned something about the redisplay and image architecture along the way and I can polish both approaches such that they will work efficiently. I can transfer code back and forth between the two approaches. All that differs is the exposed API.

1. The most minimal canvas API via image specifications. A canvas is simply an image (image :canvas-id ...) and the underlying memory can be accessed via canvas_pixel. The whole drawing must be implemented in a native module. I like the minimalism of the API and the way it is integrated coherently with the image API. But it is not as useful out of the box on the Lisp level. My latest patch uses this approach and is already polished.

2. The canvas API with exposed canvas objects. This API provides more or less canvas-make, canvas-refresh, canvas-from-image and canvas-compose to copy canvases on top of each other with blending. One can also access the memory via canvas_pixel but the API is also usable from Lisp. I like the composition aspect, but providing more drawing operations seems out of scope since then we would replicate other drawing libraries. One can still use this for drawing by passing SVGs to canvas-from-image and then blend the result in the destination canvas. Besides that the drawing capabilities are still restricted, and the SVG approach feels hacky.

The discussion on the Emacs bug tracker did not exactly go well - it sounded more as if the proposal is going to be rejected, given the basic questions that were being asked. This seems rudimentary? Why is this even useful? Why not add Emacs reader directly to Emacs? This means that it is not clear which approach is even desirable in core.

The minimal approach might be all that we need to get the ball rolling and implement richer canvas facilities outside of the core, and also not commit right away on how things are actually drawn (via GL, via some other lib, ...).

The minimal approach should be completely sufficient for use cases like Emacs reader or games like Doom, since all the pixel data is coming from outside. For Emacs reader one still needs some drawing operations for annotations and highlighting but this is secondary. For custom widget drawing purely in Lisp, approach 2 might be better, but then, it may even fall short, since drawing is still too restricted and everything has to go through SVG subcanvases which are then composed.

But I should say that nothing is set in stone, and maybe the canvas API won't make it into Emacs. When I think about the canvas API I see two approaches, the minimal one and a slightly more sophisticated one. I learned something about the redisplay and image architecture along the way and I can polish both approaches such that they will work efficiently. I can transfer code back and forth between the two approaches. All that differs is the exposed API. 1. The most minimal canvas API via image specifications. A canvas is simply an image `(image :canvas-id ...)` and the underlying memory can be accessed via `canvas_pixel`. The whole drawing must be implemented in a native module. I like the minimalism of the API and the way it is integrated coherently with the image API. But it is not as useful out of the box on the Lisp level. My latest patch uses this approach and is already polished. 2. The canvas API with exposed canvas objects. This API provides more or less `canvas-make`, `canvas-refresh`, `canvas-from-image` and `canvas-compose` to copy canvases on top of each other with blending. One can also access the memory via `canvas_pixel` but the API is also usable from Lisp. I like the composition aspect, but providing more drawing operations seems out of scope since then we would replicate other drawing libraries. One can still use this for drawing by passing SVGs to `canvas-from-image` and then blend the result in the destination canvas. Besides that the drawing capabilities are still restricted, and the SVG approach feels hacky. The discussion on the Emacs bug tracker did not exactly go well - it sounded more as if the proposal is going to be rejected, given the basic questions that were being asked. This seems rudimentary? Why is this even useful? Why not add Emacs reader directly to Emacs? This means that it is not clear which approach is even desirable in core. The minimal approach might be all that we need to get the ball rolling and implement richer canvas facilities outside of the core, and also not commit right away on how things are actually drawn (via GL, via some other lib, ...). The minimal approach should be completely sufficient for use cases like Emacs reader or games like Doom, since all the pixel data is coming from outside. For Emacs reader one still needs some drawing operations for annotations and highlighting but this is secondary. For custom widget drawing purely in Lisp, approach 2 might be better, but then, it may even fall short, since drawing is still too restricted and everything has to go through SVG subcanvases which are then composed.

divyaranjan commented 2026年02月07日 15:18:06 +01:00

@minad wrote in #10 (comment):

But I should say that nothing is set in stone, and maybe the canvas API won't make it into Emacs. When I think about the canvas API I see two approaches, the minimal one and a slightly more sophisticated one. I learned something about the redisplay and image architecture along the way and I can polish both approaches such that they will work efficiently. I can transfer code back and forth between the two approaches. All that differs is the exposed API.

1. The most minimal canvas API via image specifications. A canvas is simply an image `(image :canvas-id ...)` and the underlying memory can be accessed via `canvas_pixel`. The whole drawing must be implemented in a native module. I like the minimalism of the API and the way it is integrated coherently with the image API. But it is not as useful out of the box on the Lisp level. My latest patch uses this approach and is already polished.
2. The canvas API with exposed canvas objects. This API provides more or less `canvas-make`, `canvas-refresh`, `canvas-from-image` and `canvas-compose` to copy canvases on top of each other with blending. One can also access the memory via `canvas_pixel` but the API is also usable from Lisp. I like the composition aspect, but providing more drawing operations seems out of scope since then we would replicate other drawing libraries. One can still use this for drawing by passing SVGs to `canvas-from-image` and then blend the result in the destination canvas. Besides that the drawing capabilities are still restricted, and the SVG approach feels hacky.

The discussion on the Emacs bug tracker did not exactly go well - it sounded more as if the proposal is going to be rejected, given the basic questions that were being asked. This seems rudimentary? Why is this even useful? Why not add Emacs reader directly to Emacs? This means that it is not clear which approach is even desirable in core.

Do you think we lacked in our explanation somewhere, that we can do better?

The minimal approach might be all that we need to get the ball rolling and implement richer canvas facilities outside of the core, and also not commit right away on how things are actually drawn (via GL, via some other lib, ...).

The minimal approach should be completely sufficient for use cases like Emacs reader or games like Doom, since all the pixel data is coming from outside. For Emacs reader one still needs some drawing operations for annotations and highlighting but this is secondary. For custom widget drawing purely in Lisp, approach 2 might be better, but then, it may even fall short, since drawing is still too restricted and everything has to go through SVG subcanvases which are then composed.

The core API can be as minimal as you can make it, as long as we have a pixel buffer exposed through a dynamic module, Pale can recreate the same primitives (and expose them to Lisp) in the library. I don't understand why we'd need SVG though? We'll simply have primitivs like canvas-rect, canvas-circle, and on top of it operations that do stuff like round the corners of the rectangle and so on. We can do all of this outside of Emacs with just raw pixels, either with software rendering or GPU. Essentially we'll have a drawing library of our own designed specifically for Emacs.

My rough goal with Pale is to have it be the equivalent of SDL but for Emacs drawing, both as a C library that somebody can use in their own dynamic module and as an Elisp library too.

I am fine with either of the Canvas APIs, while I personally would prefer the second one, I'm not too hung up on that. I think what's more important is we have an auxillary library outside of Emacs that can take this functionality and make good use of it.

@minad wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10426592: > But I should say that nothing is set in stone, and maybe the canvas API won't make it into Emacs. When I think about the canvas API I see two approaches, the minimal one and a slightly more sophisticated one. I learned something about the redisplay and image architecture along the way and I can polish both approaches such that they will work efficiently. I can transfer code back and forth between the two approaches. All that differs is the exposed API. > > 1. The most minimal canvas API via image specifications. A canvas is simply an image `(image :canvas-id ...)` and the underlying memory can be accessed via `canvas_pixel`. The whole drawing must be implemented in a native module. I like the minimalism of the API and the way it is integrated coherently with the image API. But it is not as useful out of the box on the Lisp level. My latest patch uses this approach and is already polished. > > 2. The canvas API with exposed canvas objects. This API provides more or less `canvas-make`, `canvas-refresh`, `canvas-from-image` and `canvas-compose` to copy canvases on top of each other with blending. One can also access the memory via `canvas_pixel` but the API is also usable from Lisp. I like the composition aspect, but providing more drawing operations seems out of scope since then we would replicate other drawing libraries. One can still use this for drawing by passing SVGs to `canvas-from-image` and then blend the result in the destination canvas. Besides that the drawing capabilities are still restricted, and the SVG approach feels hacky. > > > The discussion on the Emacs bug tracker did not exactly go well - it sounded more as if the proposal is going to be rejected, given the basic questions that were being asked. This seems rudimentary? Why is this even useful? Why not add Emacs reader directly to Emacs? This means that it is not clear which approach is even desirable in core. Do you think we lacked in our explanation somewhere, that we can do better? > The minimal approach might be all that we need to get the ball rolling and implement richer canvas facilities outside of the core, and also not commit right away on how things are actually drawn (via GL, via some other lib, ...). > > The minimal approach should be completely sufficient for use cases like Emacs reader or games like Doom, since all the pixel data is coming from outside. For Emacs reader one still needs some drawing operations for annotations and highlighting but this is secondary. For custom widget drawing purely in Lisp, approach 2 might be better, but then, it may even fall short, since drawing is still too restricted and everything has to go through SVG subcanvases which are then composed. The core API can be as minimal as you can make it, as long as we have a pixel buffer exposed through a dynamic module, Pale can recreate the same primitives (and expose them to Lisp) in the library. I don't understand why we'd need SVG though? We'll simply have primitivs like `canvas-rect`, `canvas-circle`, and on top of it operations that do stuff like round the corners of the rectangle and so on. We can do all of this outside of Emacs with just raw pixels, either with software rendering or GPU. Essentially we'll have a drawing library of our own designed specifically for Emacs. My rough goal with Pale is to have it be the equivalent of SDL but for Emacs drawing, both as a C library that somebody can use in their own dynamic module and as an Elisp library too. I am fine with either of the Canvas APIs, while I personally would prefer the second one, I'm not too hung up on that. I think what's more important is we have an auxillary library outside of Emacs that can take this functionality and make good use of it.

minad commented 2026年02月07日 15:27:19 +01:00

Do you think we lacked in our explanation somewhere, that we can do better?

Maybe. But my experience is that if the maintainers don't like something, the result is just a long discussion, which leads nowhere. Every question asked will just lead to counter arguments.

The core API can be as minimal as you can make it, as long as we have a pixel buffer exposed through a dynamic module, Pale can recreate the same primitives (and expose them to Lisp) in the library. I don't understand why we'd need SVG though? We'll simply have primitivs like canvas-rect, canvas-circle, and on top of it operations that do stuff like round the corners of the rectangle and so on. We can do all of this outside of Emacs with just raw pixels, either with software rendering or GPU. Essentially we'll have a drawing library of our own designed specifically for Emacs.

What I meant regarding SVG is that one can reuse the existing facilities in Emacs without additional drawing facilities. If there is canvas-from-image one can use this to create a pixel canvas from an SVG and then use that to draw some complex graphics.

But my preference is still to use the most minimal approach. Then all the drawing can be implemented on top. For example one could imagine an API like this:

(defun draw-begin (image)) ;; Create drawing context from canvas image descriptor
(defun draw-line (ctx x1 y2 x2 y2))
;; ...and arbitrary operations...
(defun draw-end (ctx))

My rough goal with Pale is to have it be the equivalent of SDL but for Emacs drawing, both as a C library that somebody can use in their own dynamic module and as an Elisp library too.

Sounds good. My suggestion would be to not reimplement this from scratch, but to bind to SDL or Cairo.

I am fine with either of the Canvas APIs, while I personally would prefer the second one, I'm not too hung up on that. I think what's more important is we have an auxillary library outside of Emacs that can take this functionality and make good use of it.

Why? I think it is the worse API since it exposes a half-baked canvas object which cannot be used for many things, and it is not clear which canvas operations to provide, as I tried to explain above. Furthermore the API seems more difficult to use, since you have to distinguish the canvas object and the image specification using the canvas. For the minimal API you just have to handle one object, the image specification, which references the canvas by id. This is at least my experience with DOOM and my GL demo.

> Do you think we lacked in our explanation somewhere, that we can do better? Maybe. But my experience is that if the maintainers don't like something, the result is just a long discussion, which leads nowhere. Every question asked will just lead to counter arguments. > The core API can be as minimal as you can make it, as long as we have a pixel buffer exposed through a dynamic module, Pale can recreate the same primitives (and expose them to Lisp) in the library. I don't understand why we'd need SVG though? We'll simply have primitivs like canvas-rect, canvas-circle, and on top of it operations that do stuff like round the corners of the rectangle and so on. We can do all of this outside of Emacs with just raw pixels, either with software rendering or GPU. Essentially we'll have a drawing library of our own designed specifically for Emacs. What I meant regarding SVG is that one can reuse the existing facilities in Emacs without additional drawing facilities. If there is `canvas-from-image` one can use this to create a pixel canvas from an SVG and then use that to draw some complex graphics. But my preference is still to use the most minimal approach. Then all the drawing can be implemented on top. For example one could imagine an API like this: ~~~ elisp (defun draw-begin (image)) ;; Create drawing context from canvas image descriptor (defun draw-line (ctx x1 y2 x2 y2)) ;; ...and arbitrary operations... (defun draw-end (ctx)) ~~~ > My rough goal with Pale is to have it be the equivalent of SDL but for Emacs drawing, both as a C library that somebody can use in their own dynamic module and as an Elisp library too. Sounds good. My suggestion would be to not reimplement this from scratch, but to bind to SDL or Cairo. > I am fine with either of the Canvas APIs, while I personally would prefer the second one, I'm not too hung up on that. I think what's more important is we have an auxillary library outside of Emacs that can take this functionality and make good use of it. Why? I think it is the worse API since it exposes a half-baked canvas object which cannot be used for many things, and it is not clear which canvas operations to provide, as I tried to explain above. Furthermore the API seems more difficult to use, since you have to distinguish the canvas object and the image specification using the canvas. For the minimal API you just have to handle one object, the image specification, which references the canvas by id. This is at least my experience with DOOM and my GL demo.

divyaranjan commented 2026年02月07日 15:39:43 +01:00

@minad wrote in #10 (comment):

Do you think we lacked in our explanation somewhere, that we can do better?

Maybe. But my experience is that if the maintainers don't like something, the result is just a long discussion, which leads nowhere. Every question asked will just lead to counter arguments.

That is rather sad.

The core API can be as minimal as you can make it, as long as we have a pixel buffer exposed through a dynamic module, Pale can recreate the same primitives (and expose them to Lisp) in the library. I don't understand why we'd need SVG though? We'll simply have primitivs like canvas-rect, canvas-circle, and on top of it operations that do stuff like round the corners of the rectangle and so on. We can do all of this outside of Emacs with just raw pixels, either with software rendering or GPU. Essentially we'll have a drawing library of our own designed specifically for Emacs.

What I meant regarding SVG is that one can reuse the existing facilities in Emacs without additional drawing facilities. If there is canvas-from-image one can use this to create a pixel canvas from an SVG and then use that to draw some complex graphics.

But my preference is still to use the most minimal approach. Then all the drawing can be implemented on top. For example one could imagine an API like this:

(defun draw-begin (image)) ;; Create drawing context from canvas image descriptor
(defun draw-line (ctx x1 y2 x2 y2))
;; ...and arbitrary operations...
(defun draw-end (ctx))

Yes! That is the API I have in mind as well! I don't think we should use SVG for this, it's unnecessary. This can either be in the core or in Pale. But regardless, we need it.

My rough goal with Pale is to have it be the equivalent of SDL but for Emacs drawing, both as a C library that somebody can use in their own dynamic module and as an Elisp library too.

Sounds good.

I am fine with either of the Canvas APIs, while I personally would prefer the second one, I'm not too hung up on that. I think what's more important is we have an auxillary library outside of Emacs that can take this functionality and make good use of it.

Why? I think it is the worse API since it exposes a half-baked canvas object which cannot be used for many things, and it is not clear which canvas operations to provide, as I tried to explain above. Furthermore the API seems more difficult to use, since you have to distinguish the canvas object and the image specification using the canvas. For the minimal API you just have to handle one object, the image specification, which references the canvas by id. This is at least my experience with DOOM and my GL demo.

I was under the assumption the second one would need further polishing. But the only reason I prefered it is becuase in my mind I see a canvas as "not an image", so it's sort of confusing for me to rely on image API for it. It's certainly more work to have other additional functions, but wouldn't that be cleaner? The existing image-mode API doesn't expect the image to be a pixelbuffer that'll be manipulated and changed.

But that's just my personal opinion on how the API looks, as I said before, I am more than happy as long as we have an exposed pixelbuffer. If having it as an image object makes it more likely to get merged, I'd happily support. Let perfect not be enemy of the good 😄.

@minad wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10428420: > > Do you think we lacked in our explanation somewhere, that we can do better? > > Maybe. But my experience is that if the maintainers don't like something, the result is just a long discussion, which leads nowhere. Every question asked will just lead to counter arguments. That is rather sad. > > The core API can be as minimal as you can make it, as long as we have a pixel buffer exposed through a dynamic module, Pale can recreate the same primitives (and expose them to Lisp) in the library. I don't understand why we'd need SVG though? We'll simply have primitivs like canvas-rect, canvas-circle, and on top of it operations that do stuff like round the corners of the rectangle and so on. We can do all of this outside of Emacs with just raw pixels, either with software rendering or GPU. Essentially we'll have a drawing library of our own designed specifically for Emacs. > > What I meant regarding SVG is that one can reuse the existing facilities in Emacs without additional drawing facilities. If there is `canvas-from-image` one can use this to create a pixel canvas from an SVG and then use that to draw some complex graphics. > > But my preference is still to use the most minimal approach. Then all the drawing can be implemented on top. For example one could imagine an API like this: > > ```elisp > (defun draw-begin (image)) ;; Create drawing context from canvas image descriptor > (defun draw-line (ctx x1 y2 x2 y2)) > ;; ...and arbitrary operations... > (defun draw-end (ctx)) > ``` Yes! That is the API I have in mind as well! I don't think we should use SVG for this, it's unnecessary. This can either be in the core or in Pale. But regardless, we need it. > > My rough goal with Pale is to have it be the equivalent of SDL but for Emacs drawing, both as a C library that somebody can use in their own dynamic module and as an Elisp library too. > > Sounds good. > > > I am fine with either of the Canvas APIs, while I personally would prefer the second one, I'm not too hung up on that. I think what's more important is we have an auxillary library outside of Emacs that can take this functionality and make good use of it. > > Why? I think it is the worse API since it exposes a half-baked canvas object which cannot be used for many things, and it is not clear which canvas operations to provide, as I tried to explain above. Furthermore the API seems more difficult to use, since you have to distinguish the canvas object and the image specification using the canvas. For the minimal API you just have to handle one object, the image specification, which references the canvas by id. This is at least my experience with DOOM and my GL demo. I was under the assumption the second one would need further polishing. But the only reason I prefered it is becuase in my mind I see a canvas as "not an image", so it's sort of confusing for me to rely on image API for it. It's certainly more work to have other additional functions, but wouldn't that be cleaner? The existing `image-mode` API doesn't expect the image to be a pixelbuffer that'll be manipulated and changed. But that's just my personal opinion on how the API looks, as I said before, I am more than happy as long as we have an exposed pixelbuffer. If having it as an image object makes it more likely to get merged, I'd happily support. Let perfect not be enemy of the good 😄.

minad commented 2026年02月07日 15:53:14 +01:00

@divyaranjan wrote in #10 (comment):

@minad wrote in #10 (comment):

Do you think we lacked in our explanation somewhere, that we can do better?

Maybe. But my experience is that if the maintainers don't like something, the result is just a long discussion, which leads nowhere. Every question asked will just lead to counter arguments.

That is rather sad.

Maybe my interpretation is clouded a bit by past experiences, but when I read through the feature request, there wasn't really a single response from the maintainers where I see a positive connotation. Maybe your interpretation is different and we just have to present a better patch? For example my minimal canvas.diff patch is pretty much polished. On the plus side, there were some enthusiastic responses by other people.

But my preference is still to use the most minimal approach. Then all the drawing can be implemented on top. For example one could imagine an API like this:

(defun draw-begin (image)) ;; Create drawing context from canvas image descriptor
(defun draw-line (ctx x1 y2 x2 y2))
;; ...and arbitrary operations...
(defun draw-end (ctx))

Yes! That is the API I have in mind as well! I don't think we should use SVG for this, it's unnecessary. This can either be in the core or in Pale. But regardless, we need it.

So we agree on this one :)

I am fine with either of the Canvas APIs, while I personally would prefer the second one, I'm not too hung up on that. I think what's more important is we have an auxillary library outside of Emacs that can take this functionality and make good use of it.

Why? I think it is the worse API since it exposes a half-baked canvas object which cannot be used for many things, and it is not clear which canvas operations to provide, as I tried to explain above. Furthermore the API seems more difficult to use, since you have to distinguish the canvas object and the image specification using the canvas. For the minimal API you just have to handle one object, the image specification, which references the canvas by id. This is at least my experience with DOOM and my GL demo.

I was under the assumption the second one would need further polishing. But the only reason I prefered it is becuase in my mind I see a canvas as "not an image", so it's sort of confusing for me to rely on image API for it. It's certainly more work to have other additional functions, but wouldn't that be cleaner? The existing image-mode API doesn't expect the image to be a pixelbuffer that'll be manipulated and changed.

Yes, the second one is not yet polished as much as the first one. The reason why I am using an image is that the canvas is really just an image in memory. This way the canvas can be handled precisely like other images in the display engine. The display property already supports images, and so we only add another type there.

To be clear, behind the image descriptor there is a canvas object which holds the memory, but this object is not exposed on the Lisp level or on the level of the dynamic module.

But that's just my personal opinion on how the API looks, as I said before, I am more than happy as long as we have an exposed pixelbuffer. If having it as an image object makes it more likely to get merged, I'd happily support. Let perfect not be enemy of the good 😄.

I don't know what is more likely. But I still don't understand why you prefer the separate canvas object. I find this less elegant since we have basically a separate opaque handle object, with which we cannot do anything, if we create an equivalent API. In any case, I find it important to discuss this and figure out what we actually want.

Compare these:

;; Approach 1
(defvar canvas-image `(image :canvas-id my-canvas :canvas-width ,w :canvas-height ,h))
(insert (propertize "#" 'display canvas-image))
(canvas-refresh canvas-image)
env->canvas_pixel(env, canvas-image)

;; Approach 2
(defvar canvas (make-image w h))
(defvar canvas-image `(image :canvas ,canvas))
(insert (propertize "#" 'display canvas-image))
(canvas-refresh canvas)
env->canvas_pixel(env, canvas)

Now things would be different if we could use the canvas directly like this:

;; Approach 3
(defvar canvas (make-canvas w h))
(insert (propertize "#" 'display canvas))
(canvas-refresh canvas)
env->canvas_pixel(env, canvas)

But approach 3 would be much more intrusive change in Emacs and it would be a huge waste. In fact, I had considered it first, and then concluded that I am just reimplementing images for no good reason. I would have to duplicate all the image features again. So the design progression in my patches was this: approach 3 -> approach 2 -> approach 1. The result is that the exposed API on the Lisp level is as minimal as possible. Maybe this explains the design and my current opinion a bit better?

@divyaranjan wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10428636: > @minad wrote in #10 (comment): > > > > Do you think we lacked in our explanation somewhere, that we can do better? > > > > > > Maybe. But my experience is that if the maintainers don't like something, the result is just a long discussion, which leads nowhere. Every question asked will just lead to counter arguments. > > That is rather sad. Maybe my interpretation is clouded a bit by past experiences, but when I read through the feature request, there wasn't really a single response from the maintainers where I see a positive connotation. Maybe your interpretation is different and we just have to present a better patch? For example my minimal canvas.diff patch is pretty much polished. On the plus side, there were some enthusiastic responses by other people. > > But my preference is still to use the most minimal approach. Then all the drawing can be implemented on top. For example one could imagine an API like this: > > ```elisp > > (defun draw-begin (image)) ;; Create drawing context from canvas image descriptor > > (defun draw-line (ctx x1 y2 x2 y2)) > > ;; ...and arbitrary operations... > > (defun draw-end (ctx)) > > ``` > > Yes! That is the API I have in mind as well! I don't think we should use SVG for this, it's unnecessary. This can either be in the core or in Pale. But regardless, we need it. So we agree on this one :) > > > I am fine with either of the Canvas APIs, while I personally would prefer the second one, I'm not too hung up on that. I think what's more important is we have an auxillary library outside of Emacs that can take this functionality and make good use of it. > > > > > > Why? I think it is the worse API since it exposes a half-baked canvas object which cannot be used for many things, and it is not clear which canvas operations to provide, as I tried to explain above. Furthermore the API seems more difficult to use, since you have to distinguish the canvas object and the image specification using the canvas. For the minimal API you just have to handle one object, the image specification, which references the canvas by id. This is at least my experience with DOOM and my GL demo. > > I was under the assumption the second one would need further polishing. But the only reason I prefered it is becuase in my mind I see a canvas as "not an image", so it's sort of confusing for me to rely on image API for it. It's certainly more work to have other additional functions, but wouldn't that be cleaner? The existing `image-mode` API doesn't expect the image to be a pixelbuffer that'll be manipulated and changed. Yes, the second one is not yet polished as much as the first one. The reason why I am using an image is that the canvas is really just an image in memory. This way the canvas can be handled precisely like other images in the display engine. The display property already supports images, and so we only add another type there. To be clear, behind the image descriptor there is a canvas object which holds the memory, but this object is not exposed on the Lisp level or on the level of the dynamic module. > But that's just my personal opinion on how the API looks, as I said before, I am more than happy as long as we have an exposed pixelbuffer. If having it as an image object makes it more likely to get merged, I'd happily support. Let perfect not be enemy of the good :smile:. I don't know what is more likely. But I still don't understand why you prefer the separate canvas object. I find this less elegant since we have basically a separate opaque handle object, with which we cannot do anything, if we create an equivalent API. In any case, I find it important to discuss this and figure out what we actually want. Compare these: ~~~ elisp ;; Approach 1 (defvar canvas-image `(image :canvas-id my-canvas :canvas-width ,w :canvas-height ,h)) (insert (propertize "#" 'display canvas-image)) (canvas-refresh canvas-image) env->canvas_pixel(env, canvas-image) ~~~ ~~~ elisp ;; Approach 2 (defvar canvas (make-image w h)) (defvar canvas-image `(image :canvas ,canvas)) (insert (propertize "#" 'display canvas-image)) (canvas-refresh canvas) env->canvas_pixel(env, canvas) ~~~ Now things would be different if we could use the canvas directly like this: ~~~ elisp ;; Approach 3 (defvar canvas (make-canvas w h)) (insert (propertize "#" 'display canvas)) (canvas-refresh canvas) env->canvas_pixel(env, canvas) ~~~ But approach 3 would be much more intrusive change in Emacs and it would be a huge waste. In fact, I had considered it first, and then concluded that I am just reimplementing images for no good reason. I would have to duplicate all the image features again. So the design progression in my patches was this: approach 3 -> approach 2 -> approach 1. The result is that the exposed API on the Lisp level is as minimal as possible. Maybe this explains the design and my current opinion a bit better?

divyaranjan commented 2026年02月08日 06:22:08 +01:00

@minad wrote in #10 (comment):

@divyaranjan wrote in #10 (comment):

@minad wrote in #10 (comment):

Do you think we lacked in our explanation somewhere, that we can do better?

Maybe. But my experience is that if the maintainers don't like something, the result is just a long discussion, which leads nowhere. Every question asked will just lead to counter arguments.

That is rather sad.

Maybe my interpretation is clouded a bit by past experiences, but when I read through the feature request, there wasn't really a single response from the maintainers where I see a positive connotation. Maybe your interpretation is different and we just have to present a better patch? For example my minimal canvas.diff patch is pretty much polished. On the plus side, there were some enthusiastic responses by other people.

Indeed, because I think others did see the point of this patch. Though Eli does seem to question the very idea of a canvas, so we need to communicate better there maybe. I haven't had the time to respond there, I'll try to do so later today.

But my preference is still to use the most minimal approach. Then all the drawing can be implemented on top. For example one could imagine an API like this:

(defun draw-begin (image)) ;; Create drawing context from canvas image descriptor
(defun draw-line (ctx x1 y2 x2 y2))
;; ...and arbitrary operations...
(defun draw-end (ctx))

Yes! That is the API I have in mind as well! I don't think we should use SVG for this, it's unnecessary. This can either be in the core or in Pale. But regardless, we need it.

So we agree on this one :)

I am fine with either of the Canvas APIs, while I personally would prefer the second one, I'm not too hung up on that. I think what's more important is we have an auxillary library outside of Emacs that can take this functionality and make good use of it.

Why? I think it is the worse API since it exposes a half-baked canvas object which cannot be used for many things, and it is not clear which canvas operations to provide, as I tried to explain above. Furthermore the API seems more difficult to use, since you have to distinguish the canvas object and the image specification using the canvas. For the minimal API you just have to handle one object, the image specification, which references the canvas by id. This is at least my experience with DOOM and my GL demo.

I was under the assumption the second one would need further polishing. But the only reason I prefered it is becuase in my mind I see a canvas as "not an image", so it's sort of confusing for me to rely on image API for it. It's certainly more work to have other additional functions, but wouldn't that be cleaner? The existing image-mode API doesn't expect the image to be a pixelbuffer that'll be manipulated and changed.

Yes, the second one is not yet polished as much as the first one. The reason why I am using an image is that the canvas is really just an image in memory. This way the canvas can be handled precisely like other images in the display engine. The display property already supports images, and so we only add another type there.

To be clear, behind the image descriptor there is a canvas object which holds the memory, but this object is not exposed on the Lisp level or on the level of the dynamic module.

But that's just my personal opinion on how the API looks, as I said before, I am more than happy as long as we have an exposed pixelbuffer. If having it as an image object makes it more likely to get merged, I'd happily support. Let perfect not be enemy of the good 😄.

I don't know what is more likely. But I still don't understand why you prefer the separate canvas object. I find this less elegant since we have basically a separate opaque handle object, with which we cannot do anything, if we create an equivalent API. In any case, I find it important to discuss this and figure out what we actually want.

Compare these:

;; Approach 1
(defvar canvas-image `(image :canvas-id my-canvas :canvas-width ,w :canvas-height ,h))
(insert (propertize "#" 'display canvas-image))
(canvas-refresh canvas-image)
env->canvas_pixel(env, canvas-image)

;; Approach 2
(defvar canvas (make-image w h))
(defvar canvas-image `(image :canvas ,canvas))
(insert (propertize "#" 'display canvas-image))
(canvas-refresh canvas)
env->canvas_pixel(env, canvas)

Now things would be different if we could use the canvas directly like this:

;; Approach 3
(defvar canvas (make-canvas w h))
(insert (propertize "#" 'display canvas))
(canvas-refresh canvas)
env->canvas_pixel(env, canvas)

But approach 3 would be much more intrusive change in Emacs and it would be a huge waste. In fact, I had considered it first, and then concluded that I am just reimplementing images for no good reason. I would have to duplicate all the image features again. So the design progression in my patches was this: approach 3 -> approach 2 -> approach 1. The result is that the exposed API on the Lisp level is as minimal as possible. Maybe this explains the design and my current opinion a bit better?

Right, so I think because I first saw your Approach 3 I got a bit biased towards that. I see why it is intrusive, but it feels cleaner to me since essentially it makes canvas a first-class Elisp object like an overlay. You would have to duplicae some of the features, such as scaling. but at the same time shouldn't these things be re-implemented? Or maybe not, I guess because internally canvas isn't that different from an image right?

Maybe a better approach could be to instead of introducing any notion of "canvas", just add another property like :data to all image formats. Since :data is taken, maybe :pixelbuf probably. We would have this for any image format, and also add support for it in create-image, so you could create an image object in Elisp and then also expose a pixelbuffer for that image. It is essentially your Approach 1. I think the argument we can make for it in the patch is that now we're not introducing something foreign, a "canvas", but rather just extending the image API to be applicatble for raw pixel manipulation.

@minad wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10428832: > @divyaranjan wrote in #10 (comment): > > > @minad wrote in #10 (comment): > > > > Do you think we lacked in our explanation somewhere, that we can do better? > > > > > > > > > Maybe. But my experience is that if the maintainers don't like something, the result is just a long discussion, which leads nowhere. Every question asked will just lead to counter arguments. > > > > > > That is rather sad. > > Maybe my interpretation is clouded a bit by past experiences, but when I read through the feature request, there wasn't really a single response from the maintainers where I see a positive connotation. Maybe your interpretation is different and we just have to present a better patch? For example my minimal canvas.diff patch is pretty much polished. On the plus side, there were some enthusiastic responses by other people. Indeed, because I think others did see the point of this patch. Though Eli does seem to question the very idea of a canvas, so we need to communicate better there maybe. I haven't had the time to respond there, I'll try to do so later today. > > > But my preference is still to use the most minimal approach. Then all the drawing can be implemented on top. For example one could imagine an API like this: > > > ```elisp > > > (defun draw-begin (image)) ;; Create drawing context from canvas image descriptor > > > (defun draw-line (ctx x1 y2 x2 y2)) > > > ;; ...and arbitrary operations... > > > (defun draw-end (ctx)) > > > ``` > > > > > > Yes! That is the API I have in mind as well! I don't think we should use SVG for this, it's unnecessary. This can either be in the core or in Pale. But regardless, we need it. > > So we agree on this one :) > > > > > I am fine with either of the Canvas APIs, while I personally would prefer the second one, I'm not too hung up on that. I think what's more important is we have an auxillary library outside of Emacs that can take this functionality and make good use of it. > > > > > > > > > Why? I think it is the worse API since it exposes a half-baked canvas object which cannot be used for many things, and it is not clear which canvas operations to provide, as I tried to explain above. Furthermore the API seems more difficult to use, since you have to distinguish the canvas object and the image specification using the canvas. For the minimal API you just have to handle one object, the image specification, which references the canvas by id. This is at least my experience with DOOM and my GL demo. > > > > > > I was under the assumption the second one would need further polishing. But the only reason I prefered it is becuase in my mind I see a canvas as "not an image", so it's sort of confusing for me to rely on image API for it. It's certainly more work to have other additional functions, but wouldn't that be cleaner? The existing `image-mode` API doesn't expect the image to be a pixelbuffer that'll be manipulated and changed. > > Yes, the second one is not yet polished as much as the first one. The reason why I am using an image is that the canvas is really just an image in memory. This way the canvas can be handled precisely like other images in the display engine. The display property already supports images, and so we only add another type there. > > To be clear, behind the image descriptor there is a canvas object which holds the memory, but this object is not exposed on the Lisp level or on the level of the dynamic module. > > > But that's just my personal opinion on how the API looks, as I said before, I am more than happy as long as we have an exposed pixelbuffer. If having it as an image object makes it more likely to get merged, I'd happily support. Let perfect not be enemy of the good :smile:. > > I don't know what is more likely. But I still don't understand why you prefer the separate canvas object. I find this less elegant since we have basically a separate opaque handle object, with which we cannot do anything, if we create an equivalent API. In any case, I find it important to discuss this and figure out what we actually want. > > Compare these: > > ```elisp > ;; Approach 1 > (defvar canvas-image `(image :canvas-id my-canvas :canvas-width ,w :canvas-height ,h)) > (insert (propertize "#" 'display canvas-image)) > (canvas-refresh canvas-image) > env->canvas_pixel(env, canvas-image) > ``` > > ```elisp > ;; Approach 2 > (defvar canvas (make-image w h)) > (defvar canvas-image `(image :canvas ,canvas)) > (insert (propertize "#" 'display canvas-image)) > (canvas-refresh canvas) > env->canvas_pixel(env, canvas) > ``` > > Now things would be different if we could use the canvas directly like this: > > ```elisp > ;; Approach 3 > (defvar canvas (make-canvas w h)) > (insert (propertize "#" 'display canvas)) > (canvas-refresh canvas) > env->canvas_pixel(env, canvas) > ``` > > But approach 3 would be much more intrusive change in Emacs and it would be a huge waste. In fact, I had considered it first, and then concluded that I am just reimplementing images for no good reason. I would have to duplicate all the image features again. So the design progression in my patches was this: approach 3 -> approach 2 -> approach 1. The result is that the exposed API on the Lisp level is as minimal as possible. Maybe this explains the design and my current opinion a bit better? Right, so I think because I first saw your Approach 3 I got a bit biased towards that. I see why it is intrusive, but it feels cleaner to me since essentially it makes canvas a first-class Elisp object like an overlay. You would have to duplicae some of the features, such as scaling. but at the same time shouldn't these things be re-implemented? Or maybe not, I guess because internally canvas isn't that different from an image right? Maybe a better approach could be to instead of introducing any notion of "canvas", just add another property like `:data` to all image formats. Since `:data` is taken, maybe `:pixelbuf` probably. We would have this for any image format, and also add support for it in `create-image`, so you could create an image object in Elisp and then also expose a pixelbuffer for that image. It is essentially your Approach 1. I think the argument we can make for it in the patch is that now we're not introducing something foreign, a "canvas", but rather just extending the image API to be applicatble for raw pixel manipulation.

divyaranjan commented 2026年02月08日 06:44:16 +01:00

@minad wrote in #10 (comment):

Okay, I understood for the conversion going through GPU roundtrip doesn't make sense. But specifically for video, do you think it is not needed at all to provide hardware acceleration, especially when you'd have to render high-res (say above 1920x1080) video? I wanted to keep GPU in the loop for the same reasons video players have hardware acceleration. I think we can probably do that in a better way. Any opinions on how we can do that?

I must say I am not an expert on this and I don't know exactly how video players use the gpu. I think some players rely on special hardware decoding, but this might then happen inside ffmpeg. Maybe the video data is already inside the GPU afterwards and can be transferred to display without additional roundtrips.

But if you get the image data out of ffmpeg, you have to pass it again to the GPU to transform it, e.g., distort it, apply shaders, blend, whatever. But this doesn't seem like a common use case, at least not when I watch videos.

Also note that with the Emacs canvas image API we still have to go through memory and the CPU again just to feed the data into Emacs, so this additionally diminishes the advantage of GPU usage.

Indeed, I think I'll try to investigate further into how certain libraries and applications use hardware acceleration. Probably we can find a better way than another expensive roundtrip. Texture mapping might be something to look into.

It can be sufficient enough, but should we be asking the CPU to do it, in the event that a GPU can do it better? I just don't want to have to go through the trouble in the future of having to re-integrate GPU when we'd need it, so I'd like to have some primitives ready where GPU can be exploited.

Yes, I think I prefer the CPU for simple things like blending a few rectangles in a document viewer, since it will just be simpler overall. And I think you won't win anything giving the GPU-CPU roundtrips. It certainly makes sense to measure this, depending on the use case. The situation changes of course for heavier animations, games whatever.

My suggestion would be to consider the use cases separately. Video playing, games and displaying documents are not necessarily the same use case.

Yeah, we'd probably have to think more about this, for now I'll remove GPU roundtrip for video and see how that works. I'll have try to have GPU as a separate thing and experiment with the roundtrip, measure and see where we go with it.

Anyway, I think it makes sense to experiment and try a few different approaches, before settling on a solution. The design space is quite large. From the side of Emacs I am quite happy with the simple API in my newest patch, since most design decisions are pushed outside and libraries/packages have the freedom to implement their own solution as they like. Just exposing the memory pointer is the simplest thing we can do.

Yup I am happy with the Canvas API as long as it just exposes a memory pointer.

In the patch I've added redisplay optimizations, tests and documentation. If you find time to critically read through it, I would really appreciate it, pointing out things you find unclear and so on.

Indeed I saw canvas-refresh, that is really good than manually having to call redisplay again and again. Now that with going with the approach of integrating canvas as a part of the Image API, we effectively say this is exclusively for dynamic modules. Since I only want it for dynamic modules, since I think they are the ones that can make the most out of it, but how do you think this will go with Emacs core devs?

@minad wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10425862: > > Okay, I understood for the conversion going through GPU roundtrip doesn't make sense. But specifically for video, do you think it is not needed at all to provide hardware acceleration, especially when you'd have to render high-res (say above 1920x1080) video? I wanted to keep GPU in the loop for the same reasons video players have hardware acceleration. I think we can probably do that in a better way. Any opinions on how we can do that? > > I must say I am not an expert on this and I don't know exactly how video players use the gpu. I think some players rely on special hardware decoding, but this might then happen inside ffmpeg. Maybe the video data is already inside the GPU afterwards and can be transferred to display without additional roundtrips. > > But if you get the image data out of ffmpeg, you have to pass it again to the GPU to transform it, e.g., distort it, apply shaders, blend, whatever. But this doesn't seem like a common use case, at least not when I watch videos. > > Also note that with the Emacs canvas image API we still have to go through memory and the CPU again just to feed the data into Emacs, so this additionally diminishes the advantage of GPU usage. Indeed, I think I'll try to investigate further into how certain libraries and applications use hardware acceleration. Probably we can find a better way than another expensive roundtrip. Texture mapping might be something to look into. > > It can be sufficient enough, but should we be asking the CPU to do it, in the event that a GPU can do it better? I just don't want to have to go through the trouble in the future of having to re-integrate GPU when we'd need it, so I'd like to have some primitives ready where GPU can be exploited. > > Yes, I think I prefer the CPU for simple things like blending a few rectangles in a document viewer, since it will just be simpler overall. And I think you won't win anything giving the GPU-CPU roundtrips. It certainly makes sense to measure this, depending on the use case. The situation changes of course for heavier animations, games whatever. > > My suggestion would be to consider the use cases separately. Video playing, games and displaying documents are not necessarily the same use case. Yeah, we'd probably have to think more about this, for now I'll remove GPU roundtrip for video and see how that works. I'll have try to have GPU as a separate thing and experiment with the roundtrip, measure and see where we go with it. > Anyway, I think it makes sense to experiment and try a few different approaches, before settling on a solution. The design space is quite large. From the side of Emacs I am quite happy with the simple API in my newest patch, since most design decisions are pushed outside and libraries/packages have the freedom to implement their own solution as they like. Just exposing the memory pointer is the simplest thing we can do. Yup I am happy with the Canvas API as long as it just exposes a memory pointer. > In the patch I've added redisplay optimizations, tests and documentation. If you find time to critically read through it, I would really appreciate it, pointing out things you find unclear and so on. Indeed I saw `canvas-refresh`, that is really good than manually having to call `redisplay` again and again. Now that with going with the approach of integrating canvas as a part of the Image API, we effectively say this is exclusively for dynamic modules. Since I only want it for dynamic modules, since I think they are the ones that can make the most out of it, but how do you think this will go with Emacs core devs?

minad commented 2026年02月08日 11:27:15 +01:00

Indeed, because I think others did see the point of this patch. Though Eli does seem to question the very idea of a canvas, so we need to communicate better there maybe. I haven't had the time to respond there, I'll try to do so later today.

Really this means that he plans to reject the patch, and collects counter arguments.

Maybe a better approach could be to instead of introducing any notion of "canvas", just add another property like :data to all image formats. Since :data is taken, maybe :pixelbuf probably. We would have this for any image format, and also add support for it in create-image, so you could create an image object in Elisp and then also expose a pixelbuffer for that image. It is essentially your Approach 1. I think the argument we can make for it in the patch is that now we're not introducing something foreign, a "canvas", but rather just extending the image API to be applicatble for raw pixel manipulation.

At first sight, that's not such a bad idea. However it will be very different from I am proposing, since the pixelbuf can then supposedly hold arbitrary image data with the ability to reload the data, e.g., PNG or JPEG? The problem is that this means I have to touch EVERY SINGLE image implementation in image.c, and one does not want to touch all this bad code, really. And then, we would still need an argb32 image type which will justs hold our simple format for fast transfer. So this won't fly.

(EDIT: Thinking about this again, it should be possible to reuse the existing image code to implement the reloading from the pixbuf, without touching everything. So it is a nice idea. However the problem is that the pixbuf type would be an entirely new type, essentially a byte-vector as is present in Scheme for instance. Emacs lacks such a type! I think if I were to decide it would be okay for me to introduce such a type, but it will add a lot of API, many manipulation functions. A tough sell.

Otoh Emacs already has unibyte strings, which one could use instead. So one could simply use :data set to a unibyte string, and reload from there. Then the next problem arises. Strings are usually small and GC managed, and not stable, so we cannot obtain a stable pointer. Furthermore there is no efficient access in the Emacs module API to unibyte strings. In fact, I had already asked recently to get direct string pointer access, but it was rejected. The rejection was reasonable, since the memory moves. I was aware of these implications and I would of course use the pointer only in ways which are compatible with the GC, but nevertheless, it is still too dangerous for an exposed API.)

Right, so I think because I first saw your Approach 3 I got a bit biased towards that. I see why it is intrusive, but it feels cleaner to me since essentially it makes canvas a first-class Elisp object like an overlay. You would have to duplicae some of the features, such as scaling. but at the same time shouldn't these things be re-implemented? Or maybe not, I guess because internally canvas isn't that different from an image right?

I had this impression first, too. But there are two ways how one can program Lisp - with opaque objects like such a canvas object or with plain old Lisp objects which just consist of lists, symbols, numbers and strings. This is the traditional Elisp way. The image API uses the traditional way. Images are not opaque objects but plain old Lisp objects.

With the opaque object exposed and used directly as display property, everything which is already there for images needs to be reimplemented, and just duplicated actually without much difference. Why do this then? It is simply not worth it, only to achieve some conceptual clarity.

I think it boils down to preference of opaque objects. I usually prefer them too. But Elisp is traditionally non-opaque. Look at byte compiler lambdas which one can directly inspect, or at least one could do that in old versions of Elisp. Lambdas were originally just lists like '(lambda ...), completely inspectable. I am not saying that this is good or better, but in the case of the image API, these plain old Lisp objects are what we have already, and I don't want to fight it.

Also I think it is an advantage if we do not introduce additional exposed API. That's the point of reusing image specifications. There is no additional exposed API, no additional exposed data type. Users will just have to handle the image specification and not multiple objects. This is conceptual minimalism, and I optimized for that.

My idea was that we could implement "heavier" canvas objects and drawing contexts on top of the API, e.g., draw-begin, draw-end as mentioned before, and at first introduce only a basic solution which we can work with.

Indeed, I think I'll try to investigate further into how certain libraries and applications use hardware acceleration. Probably we can find a better way than another expensive roundtrip. Texture mapping might be something to look into.

My suggestion would be to leave this for later. Investigation is fine of course.

Indeed I saw canvas-refresh, that is really good than manually having to call redisplay again and again. Now that with going with the approach of integrating canvas as a part of the Image API, we effectively say this is exclusively for dynamic modules. Since I only want it for dynamic modules, since I think they are the ones that can make the most out of it, but how do you think this will go with Emacs core devs?

It won't be accepted probably. I don't see how to properly expose this to the Lisp level without implementing a whole bunch of drawing functions, the whole crazy thing. I would prefer to write a canvas-cairo library as a module, as a separate step. Usually I tend to decouple problems, and do them step by step. I am unsure how to proceed.

I could expose a canvas-get or canvas-put function, as basic pixel drawing functions. But then I might have to expose the canvas object since otherwise we would have go through the image specification lookup every time when a pixel is drawn. Well, maybe a last_canvas EQ check would work to still make this efficient enough. But such an API was already called rudimentary, which it is, so it would be rejected too. Therefore I eliminated the API completely in my latest patches.

> Indeed, because I think others did see the point of this patch. Though Eli does seem to question the very idea of a canvas, so we need to communicate better there maybe. I haven't had the time to respond there, I'll try to do so later today. Really this means that he plans to reject the patch, and collects counter arguments. > Maybe a better approach could be to instead of introducing any notion of "canvas", just add another property like :data to all image formats. Since :data is taken, maybe :pixelbuf probably. We would have this for any image format, and also add support for it in create-image, so you could create an image object in Elisp and then also expose a pixelbuffer for that image. It is essentially your Approach 1. I think the argument we can make for it in the patch is that now we're not introducing something foreign, a "canvas", but rather just extending the image API to be applicatble for raw pixel manipulation. At first sight, that's not such a bad idea. However it will be very different from I am proposing, since the pixelbuf can then supposedly hold arbitrary image data with the ability to reload the data, e.g., PNG or JPEG? The problem is that this means I have to touch EVERY SINGLE image implementation in image.c, and one does not want to touch all this bad code, really. And then, we would still need an argb32 image type which will justs hold our simple format for fast transfer. So this won't fly. (EDIT: Thinking about this again, it should be possible to reuse the existing image code to implement the reloading from the pixbuf, without touching everything. So it is a nice idea. However the problem is that the pixbuf type would be an entirely new type, essentially a byte-vector as is present in Scheme for instance. Emacs lacks such a type! I think if I were to decide it would be okay for me to introduce such a type, but it will add a lot of API, many manipulation functions. A tough sell. Otoh Emacs already has unibyte strings, which one could use instead. So one could simply use :data set to a unibyte string, and reload from there. Then the next problem arises. Strings are usually small and GC managed, and not stable, so we cannot obtain a stable pointer. Furthermore there is no efficient access in the Emacs module API to unibyte strings. In fact, I had already asked recently to get direct string pointer access, but it was rejected. The rejection was reasonable, since the memory moves. I was aware of these implications and I would of course use the pointer only in ways which are compatible with the GC, but nevertheless, it is still too dangerous for an exposed API.) > Right, so I think because I first saw your Approach 3 I got a bit biased towards that. I see why it is intrusive, but it feels cleaner to me since essentially it makes canvas a first-class Elisp object like an overlay. You would have to duplicae some of the features, such as scaling. but at the same time shouldn't these things be re-implemented? Or maybe not, I guess because internally canvas isn't that different from an image right? I had this impression first, too. But there are two ways how one can program Lisp - with opaque objects like such a canvas object or with plain old Lisp objects which just consist of lists, symbols, numbers and strings. This is the traditional Elisp way. The image API uses the traditional way. Images are not opaque objects but plain old Lisp objects. With the opaque object exposed and used directly as display property, everything which is already there for images needs to be reimplemented, and just duplicated actually without much difference. Why do this then? It is simply not worth it, only to achieve some conceptual clarity. I think it boils down to preference of opaque objects. I usually prefer them too. But Elisp is traditionally non-opaque. Look at byte compiler lambdas which one can directly inspect, or at least one could do that in old versions of Elisp. Lambdas were originally just lists like `'(lambda ...)`, completely inspectable. I am not saying that this is good or better, but in the case of the image API, these plain old Lisp objects are what we have already, and I don't want to fight it. Also I think it is an advantage if we do not introduce additional exposed API. That's the point of reusing image specifications. There is no additional exposed API, no additional exposed data type. Users will just have to handle the image specification and not multiple objects. This is conceptual minimalism, and I optimized for that. My idea was that we could implement "heavier" canvas objects and drawing contexts on top of the API, e.g., `draw-begin`, `draw-end` as mentioned before, and at first introduce only a basic solution which we can work with. > Indeed, I think I'll try to investigate further into how certain libraries and applications use hardware acceleration. Probably we can find a better way than another expensive roundtrip. Texture mapping might be something to look into. My suggestion would be to leave this for later. Investigation is fine of course. > Indeed I saw canvas-refresh, that is really good than manually having to call redisplay again and again. Now that with going with the approach of integrating canvas as a part of the Image API, we effectively say this is exclusively for dynamic modules. Since I only want it for dynamic modules, since I think they are the ones that can make the most out of it, but how do you think this will go with Emacs core devs? It won't be accepted probably. I don't see how to properly expose this to the Lisp level without implementing a whole bunch of drawing functions, the whole crazy thing. I would prefer to write a canvas-cairo library as a module, as a separate step. Usually I tend to decouple problems, and do them step by step. I am unsure how to proceed. I could expose a canvas-get or canvas-put function, as basic pixel drawing functions. But then I might have to expose the canvas object since otherwise we would have go through the image specification lookup every time when a pixel is drawn. Well, maybe a last_canvas EQ check would work to still make this efficient enough. But such an API was already called rudimentary, which it is, so it would be rejected too. Therefore I eliminated the API completely in my latest patches.

minad commented 2026年02月08日 12:06:37 +01:00

To give an update on the patch:

I moved it from the gist to https://github.com/minad/emacs-canvas-patch, and there is also a canvas-demo.el which one can try. Of course DOOM also works with the patch. The patch uses the minimalist "approach 1" from above. I included multiple redisplay optimizations, e.g., to make sure that only the image (the single image glyph) is redrawn and to avoid updating the mode line, menu bar, tool bar, tab bar and so on on every frame, which is expensive.

I am discussing with Gerd Möllmann, who wrote the redisplay code, if there are further optimizations I could apply. But overall, he finds the approach valid modulo details, I guess. Then I am discussing with Juri, who is an experienced Emacs dev, who was somewhat excited about the patch, but said that the discussion was not convincing. I will see if something comes out of the discussion with Gerd, additional optimizations. I will then integrate these into my patch. But I am will likely not pursue the discussion on the Emacs bug tracker for the time being, until I find more time, or until there is a sign that this could be accepted.

Also I don't plan to work on a bigger canvas patch, e.g., one which provides all the drawing code directly in the core. It is most important to get access to the canvas memory buffer from Emacs modules, since otherwise we cannot use it for Emacs reader for instance. If we get a rich Lisp API, the result could still be that there is no pointer access, which misses the goal. Furthermore there are many obstacles (which library to chose, how should the API look like, ...). I would like to work on such a drawing library like Pale, e.g., SDL or Cairo bindings, but I don't want to maintain this as a large patch for Emacs for a long time, until it finally lands. I would like to write this as a separate package, since this leaves more room for experimentation. The API could still eventually land in Emacs core.

To give an update on the patch: I moved it from the gist to https://github.com/minad/emacs-canvas-patch, and there is also a canvas-demo.el which one can try. Of course DOOM also works with the patch. The patch uses the minimalist "approach 1" from above. I included multiple redisplay optimizations, e.g., to make sure that only the image (the single image glyph) is redrawn and to avoid updating the mode line, menu bar, tool bar, tab bar and so on on every frame, which is expensive. I am discussing with Gerd Möllmann, who wrote the redisplay code, if there are further optimizations I could apply. But overall, he finds the approach valid modulo details, I guess. Then I am discussing with Juri, who is an experienced Emacs dev, who was somewhat excited about the patch, but said that the discussion was not convincing. I will see if something comes out of the discussion with Gerd, additional optimizations. I will then integrate these into my patch. But I am will likely not pursue the discussion on the Emacs bug tracker for the time being, until I find more time, or until there is a sign that this could be accepted. Also I don't plan to work on a bigger canvas patch, e.g., one which provides all the drawing code directly in the core. It is most important to get access to the canvas memory buffer from Emacs modules, since otherwise we cannot use it for Emacs reader for instance. If we get a rich Lisp API, the result could still be that there is no pointer access, which misses the goal. Furthermore there are many obstacles (which library to chose, how should the API look like, ...). I would like to work on such a drawing library like Pale, e.g., SDL or Cairo bindings, but I don't want to maintain this as a large patch for Emacs for a long time, until it finally lands. I would like to write this as a separate package, since this leaves more room for experimentation. The API could still eventually land in Emacs core.

minad commented 2026年02月08日 16:35:22 +01:00

Another update:

After discussing with Gerd I've implemented a solution where the canvas glyphs are updated directly, without going through the redisplay infrastructure. Emacs doesn't update anything besides the canvas. With this DOOM takes maybe 10% CPU load at 60 FPS. Emacs could perhaps also take advantage of such single glyph updates for animated images - something which is implemented much less efficiently right now. It is pretty nice to have such single glyph update functionality, and I don't think we can do much better than that.

Another update: After discussing with Gerd I've implemented a solution where the canvas glyphs are updated directly, without going through the redisplay infrastructure. Emacs doesn't update anything besides the canvas. With this DOOM takes maybe 10% CPU load at 60 FPS. Emacs could perhaps also take advantage of such single glyph updates for animated images - something which is implemented much less efficiently right now. It is pretty nice to have such single glyph update functionality, and I don't think we can do much better than that.

divyaranjan commented 2026年02月08日 16:45:02 +01:00

@minad wrote in #10 (comment):

Another update:

After discussing with Gerd I've implemented a solution where the canvas glyphs are updated directly, without going through the redisplay infrastructure. Emacs doesn't update anything besides the canvas. With this DOOM takes maybe 10% CPU load at 60 FPS. Emacs could perhaps also take advantage of such single glyph updates for animated images - something which is implemented much less efficiently right now. It is pretty nice to have such single glyph update functionality, and I don't think we can do much better than that.

This is great news! Ping me when you update the repo with the diff, I'll start testing it immediately! And yes, we don't need to call the expensive Emacs redisplay to just update the canvas. Also good to see that Gerd has OK-ed this.

@minad wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10447298: > Another update: > > After discussing with Gerd I've implemented a solution where the canvas glyphs are updated directly, without going through the redisplay infrastructure. Emacs doesn't update anything besides the canvas. With this DOOM takes maybe 10% CPU load at 60 FPS. Emacs could perhaps also take advantage of such single glyph updates for animated images - something which is implemented much less efficiently right now. It is pretty nice to have such single glyph update functionality, and I don't think we can do much better than that. This is great news! Ping me when you update the repo with the diff, I'll start testing it immediately! And yes, we don't need to call the expensive Emacs redisplay to just update the canvas. Also good to see that Gerd has OK-ed this.

minad commented 2026年02月08日 18:22:26 +01:00

@divyaranjan Here is the update: github.com/minad/emacs-canvas-patch@665117e5f6. To be clear, I just discussed this with Gerd, and he finds that the approach can work, but this doesn't mean that he OKed this in any way.

@divyaranjan Here is the update: https://github.com/minad/emacs-canvas-patch/commit/665117e5f6da638c11284bb75d14922bd0643ce6. To be clear, I just discussed this with Gerd, and he finds that the approach can work, but this doesn't mean that he OKed this in any way.

divyaranjan commented 2026年02月14日 03:42:17 +01:00

I'm done wiring Pale with the latest Canvas API, here's a demo of Pale rendering the 30FPS video without any delay fully software rendered.

I'm done wiring Pale with the latest Canvas API, here's a demo of Pale rendering the 30FPS video without any delay fully software rendered.

divyaranjan commented 2026年02月14日 04:14:01 +01:00

And here's a full HD 1080p 60FPS video, this is nothing but pleasant. I cannot wait before I fully play videos inside Emacs all the time.

And here's a full HD 1080p 60FPS video, this is nothing but pleasant. I cannot wait before I fully play videos inside Emacs all the time.

divyaranjan commented 2026年02月14日 05:17:30 +01:00

Here's the code to render video:

(setq video-player (pale-create-canvas 1280 720 'rms-pisa))
(pale-display video-player)
(pale-video-play video-player "/home/divya/Videos/rms-pisa.mkv")

Here's the code to render video: ``` (setq video-player (pale-create-canvas 1280 720 'rms-pisa)) (pale-display video-player) (pale-video-play video-player "/home/divya/Videos/rms-pisa.mkv") ```

divyaranjan commented 2026年02月14日 05:51:06 +01:00

@minad Do you have plans of changing the API too much further? It takes a while to re-adapt all the changes.

@minad Do you have plans of changing the API too much further? It takes a while to re-adapt all the changes.

minad commented 2026年02月14日 10:51:37 +01:00

I'm done wiring Pale with the latest Canvas API, here's a demo of Pale rendering the 30FPS video without any delay fully software rendered.

Thanks for the demonstrations. This is great. Btw, my canvas patch also tries to avoid GC pressure, so ideally we won't get GC pauses. But this will be a lesser problem anyway with IGC.

Do you have plans of changing the API too much further? It takes a while to re-adapt all the changes.

No plans really.

The question is how to convince Emacs upstream to add this. Would it be possible to create a small video player we could directly add to Emacs, I mean extract it from Pale? This was one of the ideas by Juri to get the contribution accepted upstream. However I am not fond of the idea - I don't want such half-baked additions. All I want is canvas_pixel :) So the question is if it could be made small and still good enough. The upstream proposition would be videoplayer.el + videoplayer c code + my canvas patch.

> I'm done wiring Pale with the latest Canvas API, here's a demo of Pale rendering the 30FPS video without any delay fully software rendered. Thanks for the demonstrations. This is great. Btw, my canvas patch also tries to avoid GC pressure, so ideally we won't get GC pauses. But this will be a lesser problem anyway with IGC. > Do you have plans of changing the API too much further? It takes a while to re-adapt all the changes. No plans really. The question is how to convince Emacs upstream to add this. Would it be possible to create a small video player we could directly add to Emacs, I mean extract it from Pale? This was one of the ideas by Juri to get the contribution accepted upstream. However I am not fond of the idea - I don't want such half-baked additions. All I want is `canvas_pixel` :) So the question is if it could be made small and still good enough. The upstream proposition would be videoplayer.el + videoplayer c code + my canvas patch.

minad commented 2026年02月14日 10:56:46 +01:00

I'm done wiring Pale with the latest Canvas API, here's a demo of Pale rendering the 30FPS video without any delay fully software rendered.

Btw, when promoting this, I would avoid to say "software rendered". The video decoding is GPU accelerated, the video scaling to the Emacs window could be GPU accelerated (Emacs image scaling via XRENDER), 3d rendering could be GPU accelerated. All that is in software is the copying at the interface. That's the price for the simple but quite universal API.

> I'm done wiring Pale with the latest Canvas API, here's a demo of Pale rendering the 30FPS video without any delay fully software rendered. Btw, when promoting this, I would avoid to say "software rendered". The video decoding is GPU accelerated, the video scaling to the Emacs window could be GPU accelerated (Emacs image scaling via XRENDER), 3d rendering could be GPU accelerated. All that is in software is the copying at the interface. That's the price for the simple but quite universal API.

divyaranjan commented 2026年02月14日 11:06:36 +01:00

@minad wrote in #10 (comment):

I'm done wiring Pale with the latest Canvas API, here's a demo of Pale rendering the 30FPS video without any delay fully software rendered.

Thanks for the demonstrations. This is great. Btw, my canvas patch also tries to avoid GC pressure, so ideally we won't get GC pauses. But this will be a lesser problem anyway with IGC.

Indeed, this is pixel-perfect and really no glitches.

Do you have plans of changing the API too much further? It takes a while to re-adapt all the changes.

No plans really.

The question is how to convince Emacs upstream to add this. Would it be possible to create a small video player we could directly add to Emacs, I mean extract it from Pale? This was one of the ideas by Juri to get the contribution accepted upstream. However I am not fond of the idea - I don't want such half-baked additions. All I want is canvas_pixel :) So the question is if it could be made small and still good enough. The upstream proposition would be videoplayer.el + videoplayer c code + my canvas patch.

That is the big question indeed. I think one can take this out and make it independent for the core. It'd be some work, but it can be done. Do you think that would help in making our case? And yes all I need too is just canvas_pixel, it's really surprising that we have to argue so much for it. But it's okay.

@minad wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10574200: > > I'm done wiring Pale with the latest Canvas API, here's a demo of Pale rendering the 30FPS video without any delay fully software rendered. > > Thanks for the demonstrations. This is great. Btw, my canvas patch also tries to avoid GC pressure, so ideally we won't get GC pauses. But this will be a lesser problem anyway with IGC. Indeed, this is pixel-perfect and really no glitches. > > Do you have plans of changing the API too much further? It takes a while to re-adapt all the changes. > > No plans really. > > The question is how to convince Emacs upstream to add this. Would it be possible to create a small video player we could directly add to Emacs, I mean extract it from Pale? This was one of the ideas by Juri to get the contribution accepted upstream. However I am not fond of the idea - I don't want such half-baked additions. All I want is `canvas_pixel` :) So the question is if it could be made small and still good enough. The upstream proposition would be videoplayer.el + videoplayer c code + my canvas patch. That is the big question indeed. I think one can take this out and make it independent for the core. It'd be some work, but it can be done. Do you think that would help in making our case? And yes all I need too is just `canvas_pixel`, it's really surprising that we have to argue so much for it. But it's okay.

divyaranjan commented 2026年02月14日 11:08:31 +01:00

@minad wrote in #10 (comment):

I'm done wiring Pale with the latest Canvas API, here's a demo of Pale rendering the 30FPS video without any delay fully software rendered.

Btw, when promoting this, I would avoid to say "software rendered". The video decoding is GPU accelerated, the video scaling to the Emacs window could be GPU accelerated (Emacs image scaling via XRENDER), 3d rendering could be GPU accelerated. All that is in software is the copying at the interface. That's the price for the simple but quite universal API.

I had totally forgotten about that. Yeah most of the time unleses Pale is itself manipulating shaders, textures, pixels it most likely doesn't need to call the GPU. I made the current code flexible enough to turn on/off the GPU pretty easily.

@minad wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10574238: > > I'm done wiring Pale with the latest Canvas API, here's a demo of Pale rendering the 30FPS video without any delay fully software rendered. > > Btw, when promoting this, I would avoid to say "software rendered". The video decoding is GPU accelerated, the video scaling to the Emacs window could be GPU accelerated (Emacs image scaling via XRENDER), 3d rendering could be GPU accelerated. All that is in software is the copying at the interface. That's the price for the simple but quite universal API. I had totally forgotten about that. Yeah most of the time unleses Pale is itself manipulating shaders, textures, pixels it most likely doesn't need to call the GPU. I made the current code flexible enough to turn on/off the GPU pretty easily.

minad commented 2026年02月14日 11:14:13 +01:00

That is the big question indeed. I think one can take this out and make it independent for the core. It'd be some work, but it can be done. Do you think that would help in making our case? And yes all I need too is just canvas_pixel, it's really surprising that we have to argue so much for it. But it's okay.

I don't really think so. Juri has said that. The risk is that we won't end up with canvas_pixel since this is not needed for core :-P

Maybe the more sane approach would be to resend my patch (I mean the newest reworked version), with links to a few simple demos. Emacs reader, Pale video rendering, Doom running on Emacs, ... And then observe what's happening. Maybe ping Juri, Gerd or Stefan to chime in. Perhaps they want the feature and write a few words of support.

> That is the big question indeed. I think one can take this out and make it independent for the core. It'd be some work, but it can be done. Do you think that would help in making our case? And yes all I need too is just canvas_pixel, it's really surprising that we have to argue so much for it. But it's okay. I don't really think so. Juri has said that. The risk is that we won't end up with canvas_pixel since this is not needed for core :-P Maybe the more sane approach would be to resend my patch (I mean the newest reworked version), with links to a few simple demos. Emacs reader, Pale video rendering, Doom running on Emacs, ... And then observe what's happening. Maybe ping Juri, Gerd or Stefan to chime in. Perhaps they want the feature and write a few words of support.

divyaranjan commented 2026年02月14日 11:30:56 +01:00

@minad wrote in #10 (comment):

That is the big question indeed. I think one can take this out and make it independent for the core. It'd be some work, but it can be done. Do you think that would help in making our case? And yes all I need too is just canvas_pixel, it's really surprising that we have to argue so much for it. But it's okay.

I don't really think so. Juri has said that. The risk is that we won't end up with canvas_pixel since this is not needed for core :-P

Maybe the more sane approach would be to resend my patch (I mean the newest reworked version), with links to a few simple demos. Emacs reader, Pale video rendering, Doom running on Emacs, ... And then observe what's happening. Maybe ping Juri, Gerd or Stefan to chime in. Perhaps they want the feature and write a few words of support.

Indeed, Eli asked me a few things I'll try to provide a response for it and add the improvements we've made. Let's not give up until we have a definite no. The patch is minimal, non-intrusive and does absolutely the minimum of what's needed, so hopefully we'd be able to convince.

@minad wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10574370: > > That is the big question indeed. I think one can take this out and make it independent for the core. It'd be some work, but it can be done. Do you think that would help in making our case? And yes all I need too is just canvas_pixel, it's really surprising that we have to argue so much for it. But it's okay. > > I don't really think so. Juri has said that. The risk is that we won't end up with canvas_pixel since this is not needed for core :-P > > Maybe the more sane approach would be to resend my patch (I mean the newest reworked version), with links to a few simple demos. Emacs reader, Pale video rendering, Doom running on Emacs, ... And then observe what's happening. Maybe ping Juri, Gerd or Stefan to chime in. Perhaps they want the feature and write a few words of support. Indeed, Eli asked me a few things I'll try to provide a response for it and add the improvements we've made. Let's not give up until we have a definite no. The patch is minimal, non-intrusive and does absolutely the minimum of what's needed, so hopefully we'd be able to convince.

divyaranjan commented 2026年02月14日 11:32:46 +01:00

I also forked SuperTuxKart, and might play around to see if I can get Pale with canvas to work. We don't have key-release events yet, I was wondering if I could (ab)use timers for that.

I also forked SuperTuxKart, and might play around to see if I can get Pale with canvas to work. We don't have key-release events yet, I was wondering if I could (ab)use timers for that.

minad commented 2026年02月14日 12:28:36 +01:00

We don't have key-release events yet, I was wondering if I could (ab)use timers for that.

Yes, there is an open Emacs issue and patch for low level key events. In DOOM on Emacs I also have some key release heuristic, but of course it is not as good as the original DOOM behavior.

> We don't have key-release events yet, I was wondering if I could (ab)use timers for that. Yes, there is an open Emacs issue and patch for low level key events. In DOOM on Emacs I also have some key release heuristic, but of course it is not as good as the original DOOM behavior.

llcc commented 2026年02月14日 16:18:30 +01:00

Thank you all for bringing this exciting feature to Emacs. I’m not a programmer, but I truly enjoy following the discussions here and learning from your conversations.

The canvas feature was discussed previously on the mailing list [1], and you may already be aware of that thread. I’m sharing it here simply for reference.

[1] https://mail.gnu.org/r/emacs-devel/2020-04/msg01903.html

Thank you all for bringing this exciting feature to Emacs. I’m not a programmer, but I truly enjoy following the discussions here and learning from your conversations. The canvas feature was discussed previously on the mailing list [1], and you may already be aware of that thread. I’m sharing it here simply for reference. [1] https://mail.gnu.org/r/emacs-devel/2020-04/msg01903.html

minad commented 2026年02月14日 16:23:34 +01:00

@llcc Oh, I wasn't aware of this attempt. Thank you for your comment. The idea of my new canvas patch is a bit simpler, to just expose access to an image in memory which we can draw on with various drawing libraries, most importantly giving access to Emacs dynamic modules.

EDIT: The old patch is tied to the Cairo API, something which I've explicitly avoided in my patch. It should also work with just X11, and also on other platforms if ported. But the price of my new patch is that it does not offer any drawing functionality whatsoever. Drawing functionality must be implemented on top, and could take advantage of arbitrary drawing libraries (Cairo, Skia, SDL, GL, ...).

@llcc Oh, I wasn't aware of this attempt. Thank you for your comment. The idea of my new canvas patch is a bit simpler, to just expose access to an image in memory which we can draw on with various drawing libraries, most importantly giving access to Emacs dynamic modules. EDIT: The old patch is tied to the Cairo API, something which I've explicitly avoided in my patch. It should also work with just X11, and also on other platforms if ported. But the price of my new patch is that it does not offer any drawing functionality whatsoever. Drawing functionality must be implemented on top, and could take advantage of arbitrary drawing libraries (Cairo, Skia, SDL, GL, ...).

divyaranjan commented 2026年02月17日 03:14:38 +01:00

SuperTuxCart at 60FPS

SuperTuxCart at 60FPS

divyaranjan commented 2026年02月17日 03:17:53 +01:00

@divyaranjan wrote in #10 (comment):

SuperTuxCart at 60FPS

Input keys are a mess right now, pressing both of them at the same time doesn't work. We'll have to work on that. Also right now it's not using Pale. I wanted to first do it directly, but I'll port several functions here. Including the input event methods.

@divyaranjan wrote in https://codeberg.org/MonadicSheep/pale/pulls/10#issuecomment-10653845: > SuperTuxCart at 60FPS Input keys are a mess right now, pressing both of them at the same time doesn't work. We'll have to work on that. Also right now it's not using Pale. I wanted to first do it directly, but I'll port several functions here. Including the input event methods.

This pull request has changes conflicting with the target branch.