'Invalid file access property list id' on File creation [Windows with PyO3]

[wiwski]

Hello,

We are writing a program using Python and Rust with PyO3. It is working fine on Ubuntu but is throwing an exception at runtime on Windows when calling File::Create with File or FileBuilder.

Here is the line that causing issue : https://github.com/betagouv/new-aglae-data-converter/blob/40981cf1853f41cad55d6d950bdeb935036f10d8/src/converter/mod.rs#L165

The issue may or may not be directly related to hdf5-rust but maybe you will have an idea of where the issue is coming from. I looked around and tried many things (like HDF5 changing versions or building on Windows Server) but I am still facing this issue. It might be related to #195.

Let me know if you need more information. I can provide you with test data if you feel like doing some exploring and running the program.

Versions

Windows 10
rustc 1.68.2
hdf5-rust 0.8.1

Compiled against HDF5 1.10.10

Steps to compile and run

1. With HDF5 1.10.10 installed, run poetry install to install python deps
2. Build crate & make it available for python : poetry run maturin develop --release
3. Run it : poetry run .\new_aglae_data_converter\converter.py ...

Backtrace

python .\new_aglae_data_converter\converter.py -d c:\Users\witol\Downloads\lst_globals_and_std\lst_globals_and_std -o c:\Users\witol\Downloads\lst_globals_and_std -e lst globals -c c:\Users\witol\Documents\Dev\new-aglae-data-converter\lst_config.yml
INFO:__main__:Reading from : c:\Users\witol\Downloads\lst_globals_and_std\lst_globals_and_std
INFO:__main__:Saving files to : c:\Users\witol\Downloads\lst_globals_and_std
INFO:globals.converter:Starting reading files...
INFO:lst.converter:Reading from: c:\Users\witol\Downloads\lst_globals_and_std\lst_globals_and_std\lst\20230227_0002_Std_SASHI_IBA.lst
INFO:lstrs.converter:File to parse: "c:\\Users\\witol\\Downloads\\lst_globals_and_std\\lst_globals_and_std\\lst\\20230227_0002_Std_SASHI_IBA.lst"INFO:lstrs.converter:Config used: LstConfig { x: 256, y: 512, detectors: {"GAMMA": Detector { adc: 32, channels: 4096 }, "GAMMA_20": Detector { adc: 1024, channels: 4096 }, "HE1": Detector { adc: 1, channels: 2048 }, "HE2": Detector { adc: 2, channels: 2048 }, "HE3": Detector { adc: 4, channels: 2048 }, "HE4": Detector { adc: 8, channels: 2048 }, "LE0": Detector { adc: 16, channels: 2048 }, "RBS": Detector { adc: 64, channels: 512 
}}, computed_detectors: {"HE13": ["HE1", "HE2", "HE3"], "HE10": ["HE1", "HE2", "HE3", "HE4"], "HE12": ["HE3", "HE4"], "HE11": ["HE1", "HE2"]}, adcs: [1, 2, 4, 8, 16, 32, 64, 256, 512, 1024] }
INFO:lstrs.converter:Map size: MapSize { width: 1000, height: 1000, pixel_size_width: 25, pixel_size_height: 25, pen_size: 500 }
INFO:lstrs.converter:Exp info: ExpInfo { particle: "Proton", beam_energy: "3000keV", le0_filter: "40mmHe", he1_filter: "100umAl", he2_filter: "OFF", he3_filter: "100umAl", he4_filter: "100umAl" }
⠒  [00:00:12] [############################################################################################################>] 21.82 MiB/21.83 MiBINFO:lstrs.converter:H5 File path : c:\Users\witol\Downloads\lst_globals_and_std\20230227_0002_Std_SASHI_IBA.hdf5
thread '<unnamed>' panicked at 'Couldn't create the file: Invalid file access property list id: 792633534417214901', src\converter\mod.rs:163:48 
stack backtrace:
   0:     0x7ffce2a4d252 - std::backtrace_rs::backtrace::dbghelp::trace
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\std\src\..\..\backtrace\src\backtrace\dbghelp.rs:98    
   1:     0x7ffce2a4d252 - std::backtrace_rs::backtrace::trace_unsynchronized
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\std\src\..\..\backtrace\src\backtrace\mod.rs:66        
   2:     0x7ffce2a4d252 - std::sys_common::backtrace::_print_fmt
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\std\src\sys_common\backtrace.rs:65
   3:     0x7ffce2a4d252 - std::sys_common::backtrace::_print::impl$0::fmt
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\std\src\sys_common\backtrace.rs:44
   4:     0x7ffce2a633fb - core::fmt::write
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\core\src\fmt\mod.rs:1213
   5:     0x7ffce2a496ea - std::io::Write::write_fmt<std::sys::windows::stdio::Stderr>
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\std\src\io\mod.rs:1682
   6:     0x7ffce2a4cf9b - std::sys_common::backtrace::_print
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\std\src\sys_common\backtrace.rs:47
   7:     0x7ffce2a4cf9b - std::sys_common::backtrace::print
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\std\src\sys_common\backtrace.rs:34
   8:     0x7ffce2a4f8c9 - std::panicking::default_hook::closure$1
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\std\src\panicking.rs:267
   9:     0x7ffce2a4f54b - std::panicking::default_hook
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\std\src\panicking.rs:286
  10:     0x7ffce2a50160 - std::panicking::rust_panic_with_hook
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\std\src\panicking.rs:688
  11:     0x7ffce2a4feee - std::panicking::begin_panic_handler::closure$0
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\std\src\panicking.rs:579
  12:     0x7ffce2a4dc6f - std::sys_common::backtrace::__rust_end_short_backtrace<std::panicking::begin_panic_handler::closure_env$0,never$>     
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\std\src\sys_common\backtrace.rs:137
  13:     0x7ffce2a4fba0 - std::panicking::begin_panic_handler
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\std\src\panicking.rs:575
  14:     0x7ffce2a71a45 - core::panicking::panic_fmt
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\core\src\panicking.rs:64
  15:     0x7ffce2a71f66 - core::result::unwrap_failed
                               at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library\core\src\result.rs:1790
  16:     0x7ffce29e7a4d - lstrs::converter::parse_lst::h300384f279ddd499
  17:     0x7ffce29ded40 - <alloc::collections::btree::map::BTreeMap<K,V,A> as core::ops::drop::Drop>::drop::hac6246eea85e7b3c
  18:     0x7ffca455800a - PyObject_Str
  19:     0x7ffca4525c7a - PyEval_EvalFrameDefault
  20:     0x7ffca45201af - PyFunction_Vectorcall
  21:     0x7ffca4524e6d - PyEval_EvalFrameDefault
  22:     0x7ffca45201af - PyFunction_Vectorcall
  23:     0x7ffca451fa8b - PyFunction_Vectorcall
  24:     0x7ffca4528112 - PyEval_EvalFrameDefault
  25:     0x7ffca45201af - PyFunction_Vectorcall
  26:     0x7ffca45975f9 - PyEval_EvalCodeWithName
  27:     0x7ffca459753b - PyEval_EvalCodeEx
  28:     0x7ffca4597499 - PyEval_EvalCode
  29:     0x7ffca459709a - PyCodec_DecodeText
  30:     0x7ffca459701a - PyCodec_DecodeText
  31:     0x7ffca45c5097 - PyDict_DelItemString
  32:     0x7ffca45c4d3c - PyUnicode_WriteChar
  33:     0x7ffca45c590f - PyRun_SimpleFileExFlags
  34:     0x7ffca4603dac - PyRun_AnyFileExFlags
  35:     0x7ffca4603b82 - PyObject_GenericGetDict
  36:     0x7ffca45d8218 - Py_RunMain
  37:     0x7ffca45d80a1 - Py_RunMain
  38:     0x7ffca45d4b19 - Py_Main
  39:     0x7ff76c441254 - <unknown>
  40:     0x7ffd05cd7614 - BaseThreadInitThunk
  41:     0x7ffd06e626a1 - RtlUserThreadStart
Traceback (most recent call last):
  File "C:\Users\witol\Documents\Dev\new-aglae-data-converter\new_aglae_data_converter\converter.py", line 82, in <module>
    processed_files_cnt = convert(
  File "C:\Users\witol\Documents\Dev\new-aglae-data-converter\new_aglae_data_converter\converter.py", line 35, in convert
    processed_files_num += convert_lst_to_hdf5(data_path, output_path, lst_config_path)
  File "C:\Users\witol\Documents\Dev\new-aglae-data-converter\new_aglae_data_converter\lst\converter.py", line 33, in convert_lst_to_hdf5        
    lstrs.parse_lst(str(lst_file.absolute()), str(output_path.absolute()), config)
pyo3_runtime.PanicException: Couldn't create the file: Invalid file access property list id: 792633534417214901

[aldanor]

@wiwski Does this problem still occur? It would be pretty hard to help out unless you try to narrow it down, i.e. what does pyo3 have to do with it, and does it occur outside of pyo3 on the same machine in the same environment.

[trippalamb]

@aldanor I too am having this same issue though I am not using pyo3, but I am on Windows 10. It fails on the same call to create the file.

EDIT:
I'm been looking around and it comes down to the fact the below code sets b equal to H5I_ERROR_CLASS

let a:hid_t;
let b:H5I_type_t;
unsafe{
    a = H5Pcreate(*H5P_FILE_ACCESS);
    b = H5Iget_type(a);
}

And that isn't considered a valid type by FileAccess::is_valid_id_type()

I don't know enough of HDF, Rust, or C Rust interaction to be sure, but it is my guess that a shift of the H5I_type_t enum integers might be to blame.

Sure enough by switching to 1.12.2 the H5I_type_t enum returned is H5I_GENPROP_LST rather than H5I_ERROR_CLASS