error: unpack requires a buffer of 1152 bytes

[daniellevie]

Just tried to convert a PLEXOS zip file locally and got the following error.

File "C:\Users\DLEVIE\Documents\h5plexos\h5plexos\process\buildh5.py", line 198, in process_solution
value_data = list(struct.unpack('<%dd'%length, bin_file.read(8*length)))

error: unpack requires a buffer of 1152 bytes

The zip file opens without issue in the PLEXOS GUI and I am not outputting MT results.
Any idea whats causing this?

[daniellevie]

So I tested the line
value_data = list(struct.unpack('<%dd'%length, bin_file.read(8*length)))
calcsize('<%dd'%length) is equal to 1152 and 8*length is also equal to 1152 so everything should have worked according to this:
https://docs.python.org/2/library/struct.html

struct.unpack(fmt, string)

Unpack the string (presumably packed by pack(fmt, ...)) according to the given format. The result is a tuple even if it contains exactly one item. The string must contain exactly the amount of data required by the format (len(string) must equal calcsize(fmt))

Have you seen this before @GordStephen?

[GordStephen]

Very strange! No, I haven't seen this before. Have you been able to process any other PLEXOS 8.2 results successfully?

[daniellevie]

Not yet but I'm going to run another test on 8.2. The last set of results was 1 week, I'll run 1 day and see if I get a similar error.

[daniellevie]

@GordStephen the second test completed and I got the same error message.

error: unpack requires a buffer of 192 bytes

This seems to be a PLEXOS 8.2 problem. The hdf5 file is created and it can be viewed but only metadata and Interval generator data is saved. No other properties or any Year data is saved.
I spoke to @jzhan188 about this and she thinks PLEXOS could be saving extra text between the Generator and other properties in the zip file which h5plexos does not know what to do with. Is that a possibility?

The zip file and partially filled hdf5 file from the last run are here if that helps:
\nrelqnap02\PLEXOS CEII\Projects\DLevie

[GordStephen]

Yeah, it seems likely that there's some change to the binary data format in 8.2 that's causing problems. If that's true rplexos would have issues as well, could you try rplexos and see what happens there?

[daniellevie]

Yeah, it seems likely that there's some change to the binary data format in 8.2 that's causing problems. If that's true rplexos would have issues as well, could you try rplexos and see what happens there?

@bethanyfrew just tried to run rplexos on the same zip file and got the following error:

Error in db$path : $ operator not defined for this S4 class

So it looks like there might be some changes to the binary data format as you suggested. Is this something that is easy or hard to fix?

[GordStephen]

It might not be hard to fix once we know what the specific changes are, but figuring that out could take some detective work...

[bsergi]

Am curious if there is any development on this -- I ran into the same issue as @Sabre626 with some recent 8.2 results.

[GordStephen]

https://github.nrel.gov/gstephen/H5PLEXOS.jl can currently process 8.2 files, and @Sabre626 is about to commit some updates that add querying support for those files in this package (we'll close this issue once that's finalized).