codes_new_from_message does not support multi-enabled grib2 messages

[keltonhalbert]

What happened?

I'm attempting to parse a grib2 file that, unfortunately, has multi-field grib messages encoded. A copy of the grib2 file from the RAP can be found here. I know that this feature is not recommended, but unfortunately, NCEP and other data providers have not heeded such advice. For multiple reasons, I need to be able to read the file through byte streaming, as described in this documentation. However, after a significant amount of time beating my head against a wall, I cannot get eccodes to properly handle byte stream messages that have multiple fields encoded. The only way I can get this to work is when using the codes_new_from_file API, and I cannot brute-force using BytesIO objects due to #25.

The motivation behind this is to fix an issue with the Kerchunk library currently being unable to parse multi-field messages. Since this is meant for scanning and reading byte ranges from cloud data sources, it is kind of crucial to be able to handle byte streams and parse a multi-field encoded message... but I'm increasingly getting the sense that the eccodes core library doesn't support this either.

I realize this issue may be rooted in the core eccodes library that this python repo wraps, but I wanted to start here first just in case. Is there anything I can do differently to achieve the desired functionality, or is Kerchunk's desire to read grib messages through byte streaming effectively unattainable?

Edit: I forgot to mention, calling eccodes.codes_grib_multi_support_on() before eccodes.codes_new_from_message(bytes(buf)) appears to have no effect.

What are the steps to reproduce the bug?

The message bytes in question:

b'GRIB\x00\x00\x00\x02\x00\x00\x00\x00\x00\x01\x14\x9e\x00\x00\x00\x15\x01\x00\x07\x00\x00\x02\x01\x01\x07\xe5\x04\x0f\x00\x00\x00\x00\x01\x00\x0
0\x00Q\x03\x00\x00\x01\x08\x8d\x00\x00\x00\x1e\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01-\x00\x00\x00\xe1\x00\x
f8m\xa8\r\xf0sp8\x01}x@\x0f\xcb\x94@\x016\x070\x016\x070\x00@\x01}x@\x01}x@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"\x04\x00\x00\x00\x00\x02\
x02\x02\x00i\x00\x00\x00\x01\x00\x00\x00\x00g\x00\x00\x00\x00\n\xff\x00\x00\x00\x00\x00\x00\x00\x00\x17\x05\x00\x01\x08\x8d\x00(\xc1I\xa8\x8b\x80
\x04\x00\x00\t\x00\x00\xff\x00\x00\x00\x06\x06\xff\x00\x00\x88\xe1\x07\xffO\xffQ\x00)\x00\x00\x00\x00\x01-\x00\x00\x00\xe1\x00\x00\x00\x00\x00\x0
0\x00\x00\x00\x00\x01-\x00\x00\x00\xe1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x08\x01\x01\xffd\x00#\x00\x01Creator: JasPer Version 1.900.1\xffR\
x00\x0c\x00\x00\x00\x01\x00\x05\x04\x04\x00\x01\xff\\\x00\x13@HPPXPPXPPXPPXPPX\xff\x90\x00\n\x00\x00\x00\x00\x88e\x00\x01\xff\x93\xcf\xc1\\\x10\x
f6`o-\xcc@A\xdax\xa3\xff\x80\xc2Z\xa7./\xfd9"1\xbe\xb0\x12\x9bI\xa0U\xf0\xe9\xf9\x8b!\x97\r\x0bT\x99\xc8\xf7\xf6\xeb\xb5O\xd6\xfd\xb8F\x91\xcb\x9
e.\x0c\x17\x14\x96\x8e\xf6\xfb\xf5\xba\xe1\xeb\xb5\x8e\xda\xf9r\x8a0\xee\xfc\x13\xb6\xaa\x9b\xcb\xa3\x8c9\xd8\xe5\x927\x87G\xc3\xedC\x87\xda\x89\
x07\xda~\x1bV\xb6\xae\xbaP\xab\xfd\x1c}\x97?\xa8I\x9e\xefr1z\x1dB\xb4\x97\xaaTI*\xe2\xfaRMuY]\x1aI\xf3\xa3L\xdago\xcc\xe8\x0eb\xdeQ%\x12\xec\x8b\
x04"\x97\xd3\x1c\xe8rIW6\xc9\xd1\x96|\x061#/8\x9f]\xfd!\xd2p\x1cE\x0f\xa9\x9a\xfb\xe3\x10\xab\xcf\xed\xe4\xb0\xbc!\xaa\x06\xd4W\r\xeb\x05z\xa0\xe
3Z\x96\xa0\xf0\xe4\xe3s\xd0\xfcH0>\x18\x91\xd2\xd5\x92\x8e{\xa4\x92e\xab!\xa5%W\xb74\xdf|\xc5?!\x8d\xe23b\x8b\xd8\x04\xa5\x02wv\xcf\x97\xf4YE\xc2
\x1d\xe8gO|%U\xb10\xdf\xd4*\xce\xa0x\x1b\x8f\x15\xbc\xc3\xa2\x8a\xc2\xa4\xc9\x88y\xe8\xb0\xde\xb8\xe2\x9f)\xb0\xab \xd7h\xcaI\xc2\xa37[\xc7\xe1~\
xc3\xed\xb9a\xf8^\xa0-9k\xdeD\x16GOU\xb8\x1b\x94l\x0cNC\xa37\xe5\\\xb3\x9f\xd9\x89\xc647\x00\x95\xd74f~\x0bLq\x1c\xc6k\x815\x15j\x85LN\x9bn\xa2\x
d6\x91\x80\x1f\xdb\x9f\xd2l\xc1\xb5\x1a\xd6X:\xa2\xca\x17\xe9\xf1\xaf\xb7\xc7\x88\x895\xb0C<\x88J\xeb4\xfbG\x84\xde}\x8c-\xb6\xaat\x0bz\'\x1f\x94
>\xd3l\xbe\xed}\xc4\xcd\xc3_\xdd\x84eUh\x92\xffz1\xc2\xc9\xf3"\xc6\xb2J\xee:\xc0Izz\xf4N3\x02\xf6\x16\xb8\x15\'B\x8d,1\xe1\x9b!\xd6\x9c\xd5VW)o\x
1eF\xbc\xefZ\xef\xe0\xb0j\xdf\x94\xa8\xa4|Q\x1a\x8f\xa9\x82\xdfvv\xb5\x11\xa5m]\x1d~:\xc5Q\xc6\xfb\x1d\xcb!\xe5\x98\xf8t8\x06\x8c\xe7\x8fj\xcb%\x
9a\xe9\xbe\x15\x95Ph\x04<A \t\x0b\x06\xec\xbcmj\x95\xfbk\xd76Y\xcf\xd2\xb0\x86\xaa\xd0\xc1\t\xf6\xa1\xd3\xbe{\x0e\x92\xca\xb7\xf9\x1e1\ts\x8d\xde
XCH\xff\x7fR\xe7\xf3Te\xd1c{B\xfck}4\xea.\x88\x10}\xc3?\x8e\xa3X\x97\xd8x>\xaed\xd0\xf0Q\xf8\x8d\x08\x1c\x8f\x19)\xaf}\x9f\xf6\xee\xe0iJ\xf0\xe2\
x11\xc0l&\xf9\xad9z9zS\xad\xdf\xcc\x7fX?\x98\x90\xac\xf5\xf1_~K\x83\'\xe5\x01\xdcUK6\x0b\xc1@\xc0\xc6i\x8fR\x00\xdf\x88\x06P*\xa6\xf4\x0b\x98\xd3
u\xf1\r\xc0\r\x08\xa2~K\x9e\xfa|Z\x80\x88\xf6u\x88G\xb6\xc0\xbc\xc2Oe\x97\x89\xf3\xa6\xc1\xd8\xe9-=M#\xc6o\xeff%Ds\xe8\xb9['

Version

2.36.0

Platform (OS and architecture)

Linux DESKTOP-5DTCKL2.attlocal.net 6.9.12-100.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Jul 27 16:09:11 UTC 2024 x86_64 GNU/Linux

Relevant log output

No response

Accompanying data

No response

Organisation

NOAA

[shahramn]

You could convert the multi-field GRIB file to a normal single-field GRIB via the ecCodes command:
grib_copy multi.grib single.grib

See https://confluence.ecmwf.int/display/UDOC/How+can+I+convert+a+multi-field+GRIB+file+to+single-field+-+ecCodes+GRIB+FAQ