what is the optimal workflow to encode bufferedimage animations of TYPE_INT_ARGB (alpha composite)?

[istinnstudio]

Hi, this is my first attempt to use this excellent library. I need to create high resolution videos from bufferedimage animations (timers in paintcomponent), with extensive use of alpha transparency by using alpha composite and custom composite objects. Bufferedimages are of TYPE_INT_ARGB. I was wondering what is the optimal way to encode this animation with ffmpeg to a high quality video. Nvidia RTX GPU acceleration would be great if it could be used. As far as I can see there is a setPixelFormat(AV_PIX_FMT_RGBA); that sets the pixel format the RGBA realm, is that OK to use, is it optimal? Do I have to use this to have the highest encoding quality? Do I have to use a specific encoder as an intermediaten video and then use the ("h264_nvenc") GPU-accelerated codec for the final export? Confused! Thanks...

[saudet]

If you're saying you want to encode in RGBA, you'll need to select manually a codec that supports that pixel format, yes, but that's it. Not many do support that though.

[saudet]

We can call setPixelFormat() and record() with different pixel formats. Conversion will happen, that's what it's for.

[istinnstudio]

that is the way I use it. setPixelFormat(AV_PIX_FMT_RGBA) this one produces color with the cyan/magenta/yellow tint, without setting colors are a bit different but still similar tints (the default AV_PIX_FMT_YUV420P).
setPixelFormat(AV_PIX_FMT_RGBA) gives the error above (not supported). Original bufferedimage is TYPE_INT_ARGB, there is nothing like RGBA in native java as far as I can see. For the flipChannels boolean in Java2DFrameConverter : org.bytedeco.javacv.Frame frame = converter.getFrame(image, 2.2, true); breaks everything apart even the time (speed is faster) and artifacts are enormous, quantization is obvious. Simple org.bytedeco.javacv.Frame frame = converter.convert(image) produces smooth videos as expected with correct timing but the transparency is off as we already know. So either we have an incompatibility in this combination or a bug. I wish someone could propose an encoder that also support those formats other than h264_nvenc or hevc_nvenc.

For now, I cannot see any other option other that switching alpha channel from channel 1 to channel 4 but I am not sure if this is the case here. But I need to find out a fast solution not a pixel by pixel iteration or find out at least an intermediate encoder that supports argb frame transparency before the final h264 conversion in a second step function.

[saudet]

setPixelFormat(AV_PIX_FMT_BGRA) with record(converter.getFrame(image, 1.0, true), AV_PIX_FMT_BGRA) should work correctly, but if it doesn't, please an issue about it with some sample code I can debug.

[istinnstudio]

OK I have found a solution and surprisingly is fast enough in my case. Here is the code for conversion on the BufferedImage level and used in convertImageToFrame,.

   public BufferedImage convertChannelsOnBufferedImageARGBtoRGBA(BufferedImage sourceImage) {
    int width = sourceImage.getWidth();
    int height = sourceImage.getHeight();
    
    BufferedImage targetImage = new BufferedImage(width, height, BufferedImage.TYPE_INT_ARGB);
    
    int[] sourceData = ((DataBufferInt) sourceImage.getRaster().getDataBuffer()).getData();
    int[] targetData = ((DataBufferInt) targetImage.getRaster().getDataBuffer()).getData();
    
    for (int i = 0; i < sourceData.length; i++) {
        // Assuming your sourceData is in ARGB format
        int argb = sourceData[i];
        int a = (argb >> 24) & 0xFF;
        int r = (argb >> 16) & 0xFF;
        int g = (argb >> 8) & 0xFF;
        int b = argb & 0xFF;
        
        // Reorder channels as needed
        int newArgb = (r << 24) | (g << 16) | (b << 8) | a; // Swap channels
        
        targetData[i] = newArgb;
    }
    
    return targetImage;
}

this discussion was really helpful !

[saudet]

You can copy directly to a Frame to make this faster too.

[istinnstudio]

this is how I use it, if there is a faster way please show me!

private org.bytedeco.javacv.Frame convertImageToFrame(BufferedImage image) {
      image = convertChannelsOnBufferedImageARGBtoRGBA(image);
      Java2DFrameConverter converter = new Java2DFrameConverter();
      org.bytedeco.javacv.Frame frame = converter.convert(image);
      return frame;
   }

[saudet]

Like I keep trying to tell you, FFmpeg most likely does support conversion from ARGB to RGBA or BGRA, so that's probably faster

[istinnstudio]

I see... if I go deeper to ffmpeg/javacv internal data structures and methods I might try to find out about this conversion path, there is so much you can do with images with this library, thanks for pointing that out