Trouble Reading Chill Radar Data

[mgrover1]

Opening a new issue, based on a question from @melovato

I am getting the same "error: unpack requires a buffer of 48 bytes" when reading CSU-CHILL CHL data. I have tried both pyart.io.read(filename) and pyart.io.ready_chl(filename) and the same error occurs. I am using Python 3.9 and arm_pyart 1.14.1 that I installed via conda. I can provide the package list and any other information if needed.

Thank you,
Mateo

Originally posted by @melovato in #826 (comment)

Can you share a data file that we can troubleshoot with?

[melovato]

Hello @mgrover1 and others,

Sorry for not opening up my own issue, I have been busy continuing my code (with nexrad data instead). I've included code to reproduce the issue and the error message below.

I have also tried attaching the CHL file directly but GitHub complains that it doesn't support the file type. I've then tried attaching it as a zip folder and it complains the size is too big "25 MB are allowed, 217 MB were attempted". If you have any suggestions for transferring the data, I'm all ears.

Thank you for getting back to me, I'll leave this page open and try to respond faster.
Best,
Mateo

Code to duplicate the issue:
import pyart
import os

main_path = # path to some directory for my testing and training data for a CNN
test_labels_path = os.path.join(main_path, 'test_set/labels/');
test_labels_dir = [f for f in os.listdir(test_labels_path) if f.startswith('CHL20210726_')];

numfile = 0;
filename = os.path.join(test_labels_path, test_labels_dir[numfile]);
radar = pyart.io.read(filename);

Error message I received:
runfile('/home/melovato/.config/spyder-py3/temp.py', wdir='/home/melovato/.config/spyder-py3')
Traceback (most recent call last):
File "/home/melovato/anaconda3/envs/test/lib/python3.9/site-packages/spyder_kernels/py3compat.py", line 356, in compat_exec
exec(code, globals, locals)
File "/home/melovato/.config/spyder-py3/temp.py", line 37, in
radar = pyart.io.read(filename)
File "/home/melovato/anaconda3/envs/test/lib/python3.9/site-packages/pyart/io/auto_read.py", line 116, in read
return read_chl(filename, **kwargs)
File "/home/melovato/anaconda3/envs/test/lib/python3.9/site-packages/pyart/io/chl.py", line 71, in read_chl
chl_file = ChlFile(prepare_for_read(filename))
File "/home/melovato/anaconda3/envs/test/lib/python3.9/site-packages/pyart/io/chl.py", line 249, in init
packet = self._read_block()
File "/home/melovato/anaconda3/envs/test/lib/python3.9/site-packages/pyart/io/chl.py", line 274, in _read_block
packet = self._parse_ray_hdr_block(payload)
File "/home/melovato/anaconda3/envs/test/lib/python3.9/site-packages/pyart/io/chl.py", line 344, in _parse_ray_hdr_block
packet = _unpack_structure(payload, ARCH_RAY_HEADER)
File "/home/melovato/anaconda3/envs/test/lib/python3.9/site-packages/pyart/io/chl.py", line 413, in _unpack_structure
tpl = struct.unpack(fmt, string)
error: unpack requires a buffer of 48 bytes

[mgrover1]

Can you upload to google drive, and share with my email mgroverwx@gmail.com?

[melovato]

Done, let me know if you have any issues.

[mgrover1]

Thank you!

[mgrover1]

@melovato is this happening with all of the chill files you are working with, or just a single file?

[mgrover1]

It looks like there is a mismatch in the ray header within the file. For Chill files, that part of the dataset should include the following structure:

(('azimuth', 'f'),
 ('elevation', 'f'),
 ('azimuth_width', 'f'),
 ('elevation_width', 'f'),
 ('gates', 'H'),
 ('beam_index', 'H'),
 ('ns_time', 'I'),
 ('time', 'Q'),
 ('bit_mask', 'Q'),
 ('ray_number', 'I'),
 ('num_pulses', 'I'))`

Which corresponds to a buffer size of 48 bytes.

Within the file you provided, that part of the dataset (the ray header section) is 64 bytes, indicating there is additional information included, but without knowing what additional fields are there, we cannot determine the expected buffer size.

As mentioned in this thread on stack overflow, without knowing the structure of the bytes data, we cannot unpack the data. I suspect this is either a corrupt file, or the data is not conforming to the CHILL standard.