I wonder how could that be possible.
I'd prefer to have an alpha channel rather than having to key the exported result.
I'm sure there's some ffmpeg set of options for this :)
I don't have the time now to research this, but links and info would make it easier for me to implement it :)
By running ffmpeg -pix_fmts I see all these pixel formats:
Pixel formats:
I.... = Supported Input format for conversion
.O... = Supported Output format for conversion
..H.. = Hardware accelerated format
...P. = Paletted format
....B = Bitstream format
FLAGS NAME NB_COMPONENTS BITS_PER_PIXEL
-----
IO... yuv420p 3 12
IO... yuyv422 3 16
IO... rgb24 3 24
IO... bgr24 3 24
IO... yuv422p 3 16
IO... yuv444p 3 24
IO... yuv410p 3 9
IO... yuv411p 3 12
IO... gray 1 8
IO..B monow 1 1
IO..B monob 1 1
I..P. pal8 1 8
IO... yuvj420p 3 12
IO... yuvj422p 3 16
IO... yuvj444p 3 24
..H.. xvmcmc 0 0
..H.. xvmcidct 0 0
IO... uyvy422 3 16
..... uyyvyy411 3 12
IO... bgr8 3 8
.O..B bgr4 3 4
IO... bgr4_byte 3 4
IO... rgb8 3 8
.O..B rgb4 3 4
IO... rgb4_byte 3 4
IO... nv12 3 12
IO... nv21 3 12
IO... argb 4 32
IO... rgba 4 32
IO... abgr 4 32
IO... bgra 4 32
IO... gray16be 1 16
IO... gray16le 1 16
IO... yuv440p 3 16
IO... yuvj440p 3 16
IO... yuva420p 4 20
..H.. vdpau_h264 0 0
..H.. vdpau_mpeg1 0 0
..H.. vdpau_mpeg2 0 0
..H.. vdpau_wmv3 0 0
..H.. vdpau_vc1 0 0
IO... rgb48be 3 48
IO... rgb48le 3 48
IO... rgb565be 3 16
IO... rgb565le 3 16
IO... rgb555be 3 15
IO... rgb555le 3 15
IO... bgr565be 3 16
IO... bgr565le 3 16
IO... bgr555be 3 15
IO... bgr555le 3 15
..H.. vaapi_moco 0 0
..H.. vaapi_idct 0 0
..H.. vaapi_vld 0 0
IO... yuv420p16le 3 24
IO... yuv420p16be 3 24
IO... yuv422p16le 3 32
IO... yuv422p16be 3 32
IO... yuv444p16le 3 48
IO... yuv444p16be 3 48
..H.. vdpau_mpeg4 0 0
..H.. dxva2_vld 0 0
IO... rgb444le 3 12
IO... rgb444be 3 12
IO... bgr444le 3 12
IO... bgr444be 3 12
IO... ya8 2 16
IO... bgr48be 3 48
IO... bgr48le 3 48
IO... yuv420p9be 3 13
IO... yuv420p9le 3 13
IO... yuv420p10be 3 15
IO... yuv420p10le 3 15
IO... yuv422p10be 3 20
IO... yuv422p10le 3 20
IO... yuv444p9be 3 27
IO... yuv444p9le 3 27
IO... yuv444p10be 3 30
IO... yuv444p10le 3 30
IO... yuv422p9be 3 18
IO... yuv422p9le 3 18
..H.. vda_vld 0 0
IO... gbrp 3 24
IO... gbrp9be 3 27
IO... gbrp9le 3 27
IO... gbrp10be 3 30
IO... gbrp10le 3 30
IO... gbrp16be 3 48
IO... gbrp16le 3 48
IO... yuva422p 4 24
IO... yuva444p 4 32
IO... yuva420p9be 4 22
IO... yuva420p9le 4 22
IO... yuva422p9be 4 27
IO... yuva422p9le 4 27
IO... yuva444p9be 4 36
IO... yuva444p9le 4 36
IO... yuva420p10be 4 25
IO... yuva420p10le 4 25
IO... yuva422p10be 4 30
IO... yuva422p10le 4 30
IO... yuva444p10be 4 40
IO... yuva444p10le 4 40
IO... yuva420p16be 4 40
IO... yuva420p16le 4 40
IO... yuva422p16be 4 48
IO... yuva422p16le 4 48
IO... yuva444p16be 4 64
IO... yuva444p16le 4 64
..H.. vdpau 0 0
IO... xyz12le 3 36
IO... xyz12be 3 36
..... nv16 3 16
..... nv20le 3 20
..... nv20be 3 20
IO... rgba64be 4 64
IO... rgba64le 4 64
IO... bgra64be 4 64
IO... bgra64le 4 64
IO... yvyu422 3 16
..H.. vda 0 0
I.... ya16be 2 32
I.... ya16le 2 32
IO... gbrap 4 32
IO... gbrap16be 4 64
IO... gbrap16le 4 64
..H.. qsv 0 0
..H.. mmal 0 0
..H.. d3d11va_vld 0 0
..H.. cuda 0 0
IO... 0rgb 3 24
IO... rgb0 3 24
IO... 0bgr 3 24
IO... bgr0 3 24
IO... yuv420p12be 3 18
IO... yuv420p12le 3 18
IO... yuv420p14be 3 21
IO... yuv420p14le 3 21
IO... yuv422p12be 3 24
IO... yuv422p12le 3 24
IO... yuv422p14be 3 28
IO... yuv422p14le 3 28
IO... yuv444p12be 3 36
IO... yuv444p12le 3 36
IO... yuv444p14be 3 42
IO... yuv444p14le 3 42
IO... gbrp12be 3 36
IO... gbrp12le 3 36
IO... gbrp14be 3 42
IO... gbrp14le 3 42
IO... yuvj411p 3 12
I.... bayer_bggr8 3 8
I.... bayer_rggb8 3 8
I.... bayer_gbrg8 3 8
I.... bayer_grbg8 3 8
I.... bayer_bggr16le 3 16
I.... bayer_bggr16be 3 16
I.... bayer_rggb16le 3 16
I.... bayer_rggb16be 3 16
I.... bayer_gbrg16le 3 16
I.... bayer_gbrg16be 3 16
I.... bayer_grbg16le 3 16
I.... bayer_grbg16be 3 16
IO... yuv440p10le 3 20
IO... yuv440p10be 3 20
IO... yuv440p12le 3 24
IO... yuv440p12be 3 24
IO... ayuv64le 4 64
..... ayuv64be 4 64
..H.. videotoolbox_vld 0 0
IO... p010le 3 15
IO... p010be 3 15
IO... gbrap12be 4 48
IO... gbrap12le 4 48
IO... gbrap10be 4 40
IO... gbrap10le 4 40
..H.. mediacodec 0 0
IO... gray12be 1 12
IO... gray12le 1 12
IO... gray10be 1 10
IO... gray10le 1 10
I.... p016le 3 24
I.... p016be 3 24
..H.. d3d11 0 0
IO... gray9be 1 9
IO... gray9le 1 9
..... gbrpf32be 3 96
..... gbrpf32le 3 96
..... gbrapf32be 4 128
..... gbrapf32le 4 128
..H.. drm_prime 0 0
Both rgb24 (the one I'm currently using) and rgba seem to be supported for input and output. I wonder which codec supports transparency. The documentation doesn't mention alpha or transparency a single time...
Here's a list of codecs that seem to support alpha: https://stackoverflow.com/questions/644684/turn-image-sequence-into-video-with-transparency#4372204
thank you
I've been traveling over the past weeks but will give this a shot soon :)
Hi there,
The library is great and we love de idea of feeding back the capture time (with capture.getCurrentTime()) in the code to even export intensive realtime stuff.
We try to achieve a tool that export hand recorded animations from a PGraphics to be used in a film, but we miss the ability to keep the PGraphics alpha channel.
Is it really possible with this part of the VideoExport code :
int byteNum = 0;
for (final int px : img.pixels) {
pixelsByte[byteNum++] = (byte) (px >> 16);
pixelsByte[byteNum++] = (byte) (px >> 8);
pixelsByte[byteNum++] = (byte) (px);
}
For the alpha to be present in ffmpeg, shouldn't there be a fourth value like :
pixelsByte[byteNum++] = (byte) (px >> 24);
It would be great to know, we spent hours trying to tweak the settings.json file with no result...
Thanks !
Hi! That might work. Have you tried? Do you know how to compile the library?
I think if you add the extra byte then "-pix_fmt", "rgb24" also needs to be adjusted.
I could add try that feature but I'm not sure how can I test if it works in Linux. Suggestions welcome :)
Hi thanks for quick reply,
No I haven't try yet and I don't know how to compile the library but I was going to try with javac and "-pix_fmt", "rgb24".
Is there anything special that I must know about recompiling the library ?
For Linux I have no insights, is it any different ?
Easiest might be to open the project in IntelliJ Idea and then run the ant build script.
rgb24 is 3 bytes (R G B) and you would need 4 bytes. I assume you need to use rgba instead.
For Linux... I think in general, in all OSes, playing videos with alpha can be tricky. I think they are not widely supported, but maybe I'm wrong. I don't know which video codecs support alpha, which containers support alpha. I normally use mp4 with h264. I don't know if that's an option in this case.
And about testing... I guess I need to open the video file in a video editor, place something under it and see if it's transparent. Or does VLC show the transparent parts in some special way?
Yes you are right rgb24 needs to be adjusted for something like rgb32 or rgba...
I was reading https://blender.stackexchange.com/questions/2050/any-video-formats-that-render-with-alpha-channel and wondered if the most flexible approach would be to actually export two videos: the current one in RGB and a gray scale one for the mask. What do you think about that?
I think it's a nice solution. It could be an optional feature to export an additional file with alpha extract mask, so whenever we export we could have a mask video of exactly the same duration.
Some interesting related info (kind of) : The production team I'm working with told me that when they use video files in their editing software (Adobe Premiere), files can be slower or faster to decode depending on the codec / container. Therefore when I give them .mp4 they encode it in Apple Prores before to import them.
Apple Prores has alpha ability but it's quite slow to encode (it almost worked in my tests with VideoExport but it was black with a little alpha in it, not full alpha), that's why I'm trying with the png codec but without success for now. Looking at the codecs page, an alpha extract video might be the best solution (not plenty of the alpha keyword).
I use Photoshop to check for alpha, didn't find a way with VLC. MacOs quickview also support transparency but it only work with some codec / container.
Hi everyone, I just want to let you know I succeeded at exporting video with alpha / transparency !
🥳 🎉
I tried to compile the library with Intellij Idea, foreseeing a pull request but unfortunately it didn't work because of missing dependencies.
It took me a while to try different settings and I didn't have the time to dig deeper the compilation part.
It's only a first step in making it available in this repo and we have to think about the right way to include this capability, but I'll list here what I found to make it work :
• Important and strange (took a while to figure this one out) : using a PGraphics, alpha is not kept if buffer is refreshed with pg.clear();, we must use pg.background(0,0);
• Change ffmpeg input pixel format here as :
"-pixel_format", "argb",
• Add a byte for alpha here as :
pixelsByte = new byte[img.pixelWidth * img.pixelHeight * 4];
• And here as :
for (final int px : img.pixels) {
pixelsByte[byteNum++] = (byte) (px >> 24);
pixelsByte[byteNum++] = (byte) (px >> 16);
pixelsByte[byteNum++] = (byte) (px >> 8);
pixelsByte[byteNum++] = (byte) (px);
}
• Getting rid of crf ffmpeg flag here as it's not used by some codecs.
• Codecs, here as (complete ffmpeg command) :
CMD_ENCODE_VIDEO_DEFAULT = new JSONArray(new StringList(new String[]{
// --------------- init
"[ffmpeg]",
"-y",
// --------------- input
"-f", "rawvideo",
"-vcodec", "rawvideo",
"-pixel_format", "argb",
"-s", "[width]x[height]",
"-r", "[fps]",
"-i", "-",
// --------------- output
"-an",
// --------------- codecs
// > codec : output name / container
// *************** tested successfully :
// > Png sequence : "output.mov"
"-vcodec", "png",
// > Quicktime animation : "output.mov"
// "-vcodec", "qtrle",
// > Apple prores : "output.mov"
// "-vcodec", "prores_ks",
// "-pix_fmt", "yuva444p10le",
// "-profile", "4444",
// "-alpha_bits", "8",
// "-quant_mat", "hq",
// "-q:v", "0",
// > Png images : "output-%04d.png"
// "-f", "image2",
// *************** to test (does not open in Photoshop to check for alpha) :
// > HAP : "output.mov"
// "-vcodec", "hap",
// "-format", "hap_alpha",
// > VP9 : "output.webm"
// "-vcodec", "libvpx",
// "-pix_fmt", "yuva420p",
// "-metadata:s:v:0", "alpha_mode=\"1\"",
// "-auto-alt-ref", "0",
// > DNxHR : "output.mov"
// "-vcodec", "dnxhd",
// "-profile", "dnxhr_444",
// "-pix_fmt", "yuv444p10le",
// > DNxHR : "output.mov"
// "-vcodec", "dnxhd",
// "-profile", "dnxhr_hqx",
// "-pix_fmt", "yuv422p10le",
// *************** VideoExport default, no alpha
// > h264 : "output.mp4"
// "-vcodec", "h264",
// "-pix_fmt", "yuv420p",
// "-crf", "70",
// --------------- container
"-metadata", "comment=[comment]",
"[output]"
// ---------------
}));
Hope it helps someone. Let me know if you find / test successful codec / container match working with alpha that I'm not aware of !
Hey congratulations for making it work! And thanks for sharing! I may try to merge this.
I started (but didn't finish) with the other approach: producing two video files, one RGB and one alpha.
Got a bit lost cleaning up and splitting the only java file into smaller ones :) Will share it when done.
@hamoid that's excellent !
Here a small export in gif from the Quicktime animation codec with alpha.
Nice to see someone taking a stab at this :)
I have not attempted this since I posted, but I look forward to trying again
I created a fork that can export RGB and Alpha videos separately:
https://github.com/hamoid/video_export_processing/tree/kotlinGradle
Only tested in Linux with Processing 3 and 4.
@hamoid well done 👌
I will give it a try on our project asap.
I was looking into it, and I am not sure how to make this work as you have said "I tested the library in Processing 3 and 4 and it does seem to run, and it can now generate a video pair (original video plus alpha channel video). The idea behind this was to be able to compose videos in a video editor (as layers)."
What would be the steps or which example actually can achieve this?
I run through all examples and none of them exports a video pair. I dont get alpha channel still..
@jonas-leonas When using the kotlinGradle branch of the library, there's this method you can call to enable alpha export (I assume it requires using a PGraphics to have transparency):
Thanks for fast reply! I am using processing pde 4,3.2 and using your p5test.pde. however when i open i get this problem as p5test starts with line import videoExport.*; that comes with error saying videoExport does not exist. I changed it to import com.hamoid.*; which fixes problem and exports two videos.
However I am not sure it is alpha and transparent still seing black background. is something wrong in the way I do it?
Ah thanks, I fixed that import.
I tried it and the movie-alpha.mp4 is what I expect: the brightness of the pixels represents the opacity: white is fully opaque, and black fully transparent. I think that's how most programs like Gimp or Photoshop represent alpha masks.
In your video editing program you should load both videos, and then somehow indicate that one of them is the alpha used for compositing over other video content. How this is done will depend on the video editing program used.
If you figure it out you it would be great to know how you did it :-)
In Kdenlive: I don't know if this is the best approach, but it looks ok. I used Multiply and Lumakey.
Okay, I see, thank you! So I apparently got the same result as yours. What about the output with one video instead of two? Is this possible?
Or, I guess now the task is to combine these two videos with Multiply and LumaKey combined...
Maybe it has something to do with the .mp4 format itself not allowing transparency?
I used the .mov format some time ago to have a transparent video running in a Unity app I developed for augmented reality for the HoloLens 2, when the screen needed to be transparent most of the time. The video file was huge, though.
The reason I went for 2 videos is that I couldn't find a solution that would work on different platforms and video players, nor how to test this locally. I think you can combine the two videos into one using ffmpeg. I've seen examples for this online.
Perfect, thank you! Jumping on it right away!
Hello, for my project exporting directly with alpha, I think I used this file. Maybe it still works :)
https://gist.github.com/narkai/da17716633cf33ded8baa6a0a29acada
Hey! I just figured it too and want to post my try of combining these two videos, for Processing and no Blending or Lumakey is needed afterwards, hovever the output is .mov !
importjava.io.BufferedReader;importjava.io.InputStreamReader;voidsetup(){// Define paths for FFmpeg and input/output filesStringffmpegPath="<FFMPEG_PATH_HERE>";StringinputVideo="<INPUT_VIDEO_PATH_HERE>";StringinputAlpha="<INPUT_ALPHA_VIDEO_PATH_HERE>";StringoutputVideo="<OUTPUT_VIDEO_PATH_HERE>";// Build the FFmpeg commandString[]command={ffmpegPath,"-i",inputVideo,"-i",inputAlpha,"-filter_complex","[0:v:0][1:v:0]alphamerge[out]","-map","[out]","-c:v","prores_ks","-pix_fmt","yuva444p10le",outputVideo};// Run the commandrunCommand(command);}voidrunCommand(String[]command){try{Processprocess=newProcessBuilder(command).start();// Optional: Capture and print the output of the processBufferedReaderreader=newBufferedReader(newInputStreamReader(process.getInputStream()));Stringline;while((line=reader.readLine())!=null){println(line);}// Wait for the process to completeprocess.waitFor();println("FFmpeg process completed.");}catch(Exceptione){e.printStackTrace();println("Error running FFmpeg command.");}}This was very helpful, thank you! However, I got stuck after that because at the moment I'm failing to get the right output when merging two videos into one and I need .mp4 file.
I want to be able to add a luma key in video editing software myslef to achieve transparency manually(but not from two videos that works just perfectly but from one).
Does this make sense? Could you guide me a little in that direction?
String[] command = { ffmpegPath, "-i", inputVideo, "-i", inputAlpha, "-filter_complex", "[0:v:0][1:v:0]alphamerge[out]", "-map", "[out]", "-c:v", "libx264", // libx264 for MP4 "-pix_fmt", "yuv420p", // Standard format MP4 outputVideo };
Using this method, it kind of works, but the issue is that later, when I apply the luma key in the video editing software, all the black-colored objects disappear
According to https://pixelbakery.com/recipes/video-image-formats it's not possible.
I can also say that I looked for a long time to make mp4 with alpha, and that it was just not possible.
Can you tell us a little bit about your use case ?
No due date set.
No dependencies set.
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?