Feature: Add Threading Encoding by Tile

[Legends0]

Back in 2013 a user posted this topic under the google groups discussion for openjpeg: https://groups.google.com/g/openjpeg/c/oYS8wFavNlw/m/39ZVhDaapPQJ

I've got an application that needs to do lossless JPEG-2000 encoding of very large images into 1024x1024 tiles (specifically I am only interested in encoding J2K codestreams - the codestream will be written into a NITF). I've got a fair number of cores available and will end up with a lot of tiles since the images are large; since each tile is independent, this would appear to lend itself well to parallelization but there doesn't appear to be a way in the OpenJPEG 2.0 API to do this (and I couldn't find anything obvious in the message boards that discussed a solution). Digging into the code a bit, it doesn't take much to support it though:

1. lib/openjp2/j2k.c's opj_j2k_pre_write_tile(): Rather than erroring out when p_tile_index != p_j2k->m_current_tile_number, just set p_j2k->m_current_tile_number = p_tile_index
2. Add a public flush function to openjpeg.h:

OPJ_BOOL OPJ_CALLCONV opj_flush(opj_codec_t *p_codec, opj_stream_t *p_stream)
{
opj_codec_private_t * l_codec = (opj_codec_private_t *) p_codec;
opj_stream_private_t * l_stream = (opj_stream_private_t *) p_stream;
return opj_stream_flush(p_stream, &l_codec->m_event_mgr);
}

With those changes, you can now create an opj_stream_t, opj_codec_t, and opj_image_t for each thread. For each thread, call opj_start_compress() and save off this header for one of the threads. Then start calling opj_write_tile() for each tile, doing whatever bookkeeping's necessary to stitch them together at the end. Finally call opj_end_compress() to save the footer off for one of the threads. For each of these calls, by forcing a flush with opj_flush(), you can grab off exactly the contents of the header, compressed tile, or footer in order to keep the bookkeeping straight.

I wanted to post those in case it's useful to others as well as get thoughts on how/if this could get integrated into the OpenJPEG codebase. #\2 seems pretty harmless to put in. #\1 seems like the current check is there to try to prevent people from doing something dumb by accident since if you were using a normal stream and wrote tiles out in the wrong order you'd obviously get the wrong result... either some kind of bool 'expert' flag could be added or the documentation could just make explicit that you should only call this out of order if you know what you're doing.

-Adam

This did not seem to get much attention. There has been an implementation of this request under another application as seen here in a nitro-nitf repository https://github.com/mdaus/nitro/tree/main/externals/coda-oss/modules/drivers/j2k/openjpeg

I also noticed there were some improvements to openjpeg to add multithreading support, but I am not sure the feature request brought up by Adam was addressed by #372.

Is what Adam mentioned here a feature openjpeg could implement or has it been implemented in an other way since the initial thread?