Installation of hdf5plugin

[wshobe]

We recently updated our ReEDS-2.0 instance. There is some problem with the hdf5 implementation (see below). I went through the git lfs install instructions. This is running on a Linux HPC cluster. The reeds2 conda environment seemed to build just fine. Any other diagnostics I can provide?

[For one thing, ReEDS is looking for the hdf5plugin, but the installation is not in environment.yml, and the plugin is not in the conda environment lib folder. Not sure if this is the problem, but the code does appear to be unable to find this plugin. Also, the current version of the plugin is inconsistent with some other modules in the environment, so I couldn't manually install it. ]

Here is the output from my test ReEDS run:

---

starting copy_files.py

copy_files.py | 2024-11-08 14:16:14 | INFO | Starting copy_files.py
copy_files.py | 2024-11-08 14:16:14 | INFO | Copying non-region-indexed files
copy_files.py | 2024-11-08 14:16:16 | INFO | Created 134 records
copy_files.py | 2024-11-08 14:16:16 | INFO | Created 13 records
copy_files.py | 2024-11-08 14:16:17 | INFO | Created 11 records
copy_files.py | 2024-11-08 14:16:17 | INFO | Created 18 records
copy_files.py | 2024-11-08 14:16:17 | INFO | Created 9 records
copy_files.py | 2024-11-08 14:16:17 | INFO | Created 48 records
copy_files.py | 2024-11-08 14:16:18 | INFO | Created 3 records
copy_files.py | 2024-11-08 14:16:18 | INFO | Created 1 records
copy_files.py | 2024-11-08 14:16:18 | INFO | Created 10 records
copy_files.py | 2024-11-08 14:16:18 | INFO | Created 13 records
copy_files.py | 2024-11-08 14:16:18 | INFO | Created 31 records
copy_files.py | 2024-11-08 14:16:19 | INFO | Copying region-indexed files: filtering for valid regions
copy_files.py | 2024-11-08 14:16:21 | ERROR | Traceback (most recent call last):
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "/sfs/gpfs/tardis/project/ceps-research/reeds2/runs/bill_test_Nov8_ercot_small/input_processing/copy_files.py", line 708, in
copy_files.py | 2024-11-08 14:16:21 | ERROR | df = read_file(full_path)
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "/sfs/gpfs/tardis/project/ceps-research/reeds2/runs/bill_test_Nov8_ercot_small/input_processing/ldc_prep.py", line 130, in read_file
copy_files.py | 2024-11-08 14:16:21 | ERROR | df = pd.read_hdf(filename)
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "/home/wms5f/.conda/envs/reeds2/lib/python3.11/site-packages/pandas/io/pytables.py", line 420, in read_hdf
copy_files.py | 2024-11-08 14:16:21 | ERROR | store = HDFStore(path_or_buf, mode=mode, errors=errors, **kwargs)
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "/home/wms5f/.conda/envs/reeds2/lib/python3.11/site-packages/pandas/io/pytables.py", line 579, in init
copy_files.py | 2024-11-08 14:16:21 | ERROR | self.open(mode=mode, **kwargs)
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "/home/wms5f/.conda/envs/reeds2/lib/python3.11/site-packages/pandas/io/pytables.py", line 731, in open
copy_files.py | 2024-11-08 14:16:21 | ERROR | self._handle = tables.open_file(self._path, self._mode, **kwargs)
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "/home/wms5f/.conda/envs/reeds2/lib/python3.11/site-packages/tables/file.py", line 300, in open_file
copy_files.py | 2024-11-08 14:16:21 | ERROR | return File(filename, mode, title, root_uep, filters, **kwargs)
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "/home/wms5f/.conda/envs/reeds2/lib/python3.11/site-packages/tables/file.py", line 750, in init
copy_files.py | 2024-11-08 14:16:21 | ERROR | self._g_new(filename, mode, **params)
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "tables/hdf5extension.pyx", line 484, in tables.hdf5extension.File._g_new
copy_files.py | 2024-11-08 14:16:21 | ERROR | tables.exceptions
copy_files.py | 2024-11-08 14:16:21 | ERROR | .
copy_files.py | 2024-11-08 14:16:21 | ERROR | HDF5ExtError
copy_files.py | 2024-11-08 14:16:21 | ERROR | :
copy_files.py | 2024-11-08 14:16:21 | ERROR | HDF5 error back trace
copy_files.py | 2024-11-08 14:16:21 | ERROR |
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "H5F.c", line 620, in H5Fopen
copy_files.py | 2024-11-08 14:16:21 | ERROR | unable to open file
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "H5VLcallback.c", line 3501, in H5VL_file_open
copy_files.py | 2024-11-08 14:16:21 | ERROR | failed to iterate over available VOL connector plugins
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "H5PLpath.c", line 578, in H5PL__path_table_iterate
copy_files.py | 2024-11-08 14:16:21 | ERROR | can't iterate over plugins in plugin path '(null)'
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "H5PLpath.c", line 620, in H5PL__path_table_iterate_process_path
copy_files.py | 2024-11-08 14:16:21 | ERROR | can't open directory: /home/wms5f/.conda/envs/reeds2/lib/hdf5/plugin
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "H5VLcallback.c", line 3351, in H5VL__file_open
copy_files.py | 2024-11-08 14:16:21 | ERROR | open failed
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "H5VLnative_file.c", line 97, in H5VL__native_file_open
copy_files.py | 2024-11-08 14:16:21 | ERROR | unable to open file
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "H5Fint.c", line 1990, in H5F_open
copy_files.py | 2024-11-08 14:16:21 | ERROR | unable to read superblock
copy_files.py | 2024-11-08 14:16:21 | ERROR | File "H5Fsuper.c", line 405, in H5F__super_read
copy_files.py | 2024-11-08 14:16:21 | ERROR | file signature not found
copy_files.py | 2024-11-08 14:16:21 | ERROR |
copy_files.py | 2024-11-08 14:16:21 | ERROR | End of HDF5 error back trace
copy_files.py | 2024-11-08 14:16:21 | ERROR |
copy_files.py | 2024-11-08 14:16:21 | ERROR | Unable to open/create file '/sfs/gpfs/tardis/project/ceps-research/reeds2/inputs/demand_response/evmc_shape_decrease_profile_Baseline.h5'

[kennedy-mindermann]

I've seen this issue in the past when h5 files are missing from the repo. Since you've already completed the git lfs installation, you should just need to retrieve those files from git lfs. You can do that by running the following command: git lfs fetch.

[ZekunYangCalvin]

Hi! I recently cloned the ReEDS repository, but encountered issues downloading several .h5 files managed through Git LFS. When I run runbatch.py, it seems like all file ending with .h5 are missing from the server, as I received HTTP 404 errors during the cloning process:
It seems these files are not currently available on the Git LFS server. Would it be possible for you to share the missing files via an alternative method (e.g., email, Google Drive, or another repository)? Or do I have to download them manually and pull them in the repo? Alternatively, if I need to follow a specific process to retrieve these files, please let me know.

Thank you for maintaining this valuable repository, and I appreciate your assistance in resolving this issue.

[kennedy-mindermann]

If you didn't do so already, you'll need to install Git LFS. There are instructions here: ReEDS Repository Setup

You can then try to fetch the files again with git lfs fetch.

If you're still encountering issues, I'd recommend re-cloning the repository after installing Git LFS. It would also be helpful if you could share a screenshot of the error.

[ZekunYangCalvin]

git lfs fetch

fetch: Fetching reference refs/heads/main
[f0afee97c2b03e3459aecdc6b193b4f501f9383a98e5733535d6ee2c5da2e424] Object does not exist on the server: [404] Object does not exist on the server
[9dfb263e3e6f92352a4fc1421e2e0789143620a0d096ae95269b07ce5c729b8a] Object does not exist on the server: [404] Object does not exist on the server
[2a3345b61558106e5d374062398ef260aec2caea77caa71fbb1285b0cb543767] Object does not exist on the server: [404] Object does not exist on the server
[b66b2c3f68efcc75b84fe2c5c0619898035175a8c2ae0c67dc7a5acedb35d31a] Object does not exist on the server: [404] Object does not exist on the server
[e7c708f1bb10d405d3194d064f1c3f22437cf3f5cd20c19cee1fdd30a583e37b] Object does not exist on the server: [404] Object does not exist on the server
[b8a8db7fc8571cb5655575b61c9c91ce4cd32d653285abe8358b68ef664def63] Object does not exist on the server: [404] Object does not exist on the server
[0f1d87a5b23472b4084a5c39db2070e6edd297fb09f45ddc6dac440364abe8b2] Object does not exist on the server: [404] Object does not exist on the server
[04c9a71ecd80f81d0678da5ed56fe7ae37f917525ade765e5796b7a8f73900d1] Object does not exist on the server: [404] Object does not exist on the server
[227947cdadbf47513cfd6d2a056eb3bed9508e2d4bd83d5b6fb089f084701862] Object does not exist on the server: [404] Object does not exist on the server
[908ffda22848e7e504fd304e014165d309ee1e140e0c29bae30aa723ccbef2d4] Object does not exist on the server: [404] Object does not exist on the server
[abe5229c42c7cd34aa3b91d39e9a5c034b21546f180decdfb0749b27d80c3bd2] Object does not exist on the server: [404] Object does not exist on the server
[073251244933153512cf944c59e5b498390c1c7aa9acebef2709dea22b3bd3b3] Object does not exist on the server: [404] Object does not exist on the server
[000782f04c3e789d4d88ed66a769ce184cee37ddcf72fcd37fdf60a25d343581] Object does not exist on the server: [404] Object does not exist on the server
[3c9884c1fb334a546a66bc70275728305261e3803a3fdb4b94fa1e341220e490] Object does not exist on the server: [404] Object does not exist on the server
[3ad62c4e326e8c7ab8e048478480405af3f1beda032dd128e22b8cb1c77d774b] Object does not exist on the server: [404] Object does not exist on the server
[7a77c6fa7f1b8d77ee849c092682156fe4a6801455baf6feec9113eff92846cb] Object does not exist on the server: [404] Object does not exist on the server
[000bf60e1df351bd51ce7a03849cb75cb03431cfb208490312aa8bce474007fb] Object does not exist on the server: [404] Object does not exist on the server
[f6a7b10cd977ea1438da2868d73146b2b5d18395fe143b8b478ee644c8085526] Object does not exist on the server: [404] Object does not exist on the server
[a158678c77589ed91a209e69b94422254b576f49f48deedc0fb639efd542c9ff] Object does not exist on the server: [404] Object does not exist on the server
[3a0f1df6e6833e66fb34e70e061ed5f962cf9568f3bbc6eb63c70ed05021443a] Object does not exist on the server: [404] Object does not exist on the server
[098f80b243ef734f0fd084b80c2509be4d2e993c2fb817299963afe5fb20c3fd] Object does not exist on the server: [404] Object does not exist on the server
[b72420676ec4a5b452b9af6479ef69979913f7365078af390dd974c3f848009c] Object does not exist on the server: [404] Object does not exist on the server
[678c68c6f16e9a802b413a2bfee7f1bcb1756657beba70b605f01c4719c798aa] Object does not exist on the server: [404] Object does not exist on the server
[449436407119c93c737c7ce3f34f15baf06f2abeced20525f63553ad49429923] Object does not exist on the server: [404] Object does not exist on the server
[3aa0076d8c4491d0c69fbaf8851b0b657622ce56cd2649994cb92cee832ea7c2] Object does not exist on the server: [404] Object does not exist on the server
[3e3645772d68829cd89a9716b91e84a85a051f1564ab7e276a1e4cd436c95df0] Object does not exist on the server: [404] Object does not exist on the server
[c3def685a1215670fdddee5383c00ec12ed8cee8e0f94c32f9ff0ca5342c59dc] Object does not exist on the server: [404] Object does not exist on the server
[11d908eb1fec9e5606315f8bf4318932f64ce0188217c409c4dac8ff245b6fdd] Object does not exist on the server: [404] Object does not exist on the server
[f043782b7fb2fdde766b34eba920a9b8fac68c2e58971b29db6ded9bf73f14b8] Object does not exist on the server: [404] Object does not exist on the server
[dfab7c62514745aae57d8340022f7f7736e0ed77fbd46624ce227da41f90b136] Object does not exist on the server: [404] Object does not exist on the server
[a75ce7c7f4b6dfe46f0a019d0451830ab54c2f28d1b6018f467422fc483be1c2] Object does not exist on the server: [404] Object does not exist on the server
[49aa8c7e0287c0da50302daf877473d13472071fbd6dd8a62a2554cb6bb7e059] Object does not exist on the server: [404] Object does not exist on the server
[ae9ef0271796bd6e10692f940186a99b322ec138eb0edc289c5fa20479014b91] Object does not exist on the server: [404] Object does not exist on the server
[672295725c28377623aacbbe53a6cf549f3470d3b85e3716d7cccff2c46279fe] Object does not exist on the server: [404] Object does not exist on the server
[ca56db9396247237c03c2427d53ff88ffe145bcd826f9fa72934bc56c81c1535] Object does not exist on the server: [404] Object does not exist on the server
[ee6b3612efc42fc4387b9c70e5ab340963674bb7c7efe853dce4d2980294676e] Object does not exist on the server: [404] Object does not exist on the server
[aaae0d4b5261ae0ab11da9750d4299bee6af53965cbcf03f90427b0c352da563] Object does not exist on the server: [404] Object does not exist on the server
[8bb9e36704a9dcfadb5eea3fa67d8869f935b8fcfe2ff927c697dd0522c1b666] Object does not exist on the server: [404] Object does not exist on the server
[4729004e50ffa2aba209aaaa80d5c818c6183ceffb14931e9c080b2c7786650a] Object does not exist on the server: [404] Object does not exist on the server
[c232b5736e2d281a6ed06ff90dd165642889f13ac4f133639d59b8dc66015834] Object does not exist on the server: [404] Object does not exist on the server
[9a6179c61ac5d2fc6793080b3da9c4247c0f06bc48df65c82b94a1afda101075] Object does not exist on the server: [404] Object does not exist on the server
[6e01a451f80cf2763ae218c0ec2b63693b085ec34de187309651a2b52433016f] Object does not exist on the server: [404] Object does not exist on the server
[3a84506700e9d8ee3a5d57df0a643b409fd3982c982747182e5c4a48d8475e00] Object does not exist on the server: [404] Object does not exist on the server
[351978ad2ef17210adfb7d1b6cae4224cb271ae81666f73eb7940aa7b0787f44] Object does not exist on the server: [404] Object does not exist on the server
[880a72f10965cfb780ab256a7182809b7a7002209bd9b7fa3e614f6ccfe8f454] Object does not exist on the server: [404] Object does not exist on the server
[8af086a10099f39e761b47fc8815d2c1066690afa9756a80db876f9113af6397] Object does not exist on the server: [404] Object does not exist on the server
[2038ebfa49ba76caf0b4dec6d7a522cf26dd675cc4cea45abec42b3a4342a873] Object does not exist on the server: [404] Object does not exist on the server
[4c81a1c507255ac56d07c1c2df958efd4be86130a0e6d34912610a83d2d500dc] Object does not exist on the server: [404] Object does not exist on the server
[b47be078af2adc9d5732f7e85e49d18b352b091aba277e98a9bda464e8ffb168] Object does not exist on the server: [404] Object does not exist on the server
[a5f696cb260146f30d4dfd3065219a35545059dcfa2897aa472ff893803ae9ad] Object does not exist on the server: [404] Object does not exist on the server
[9d810efeba4640883ff2fee72b3f48cc77fd154c87f079c966d7e8792daa07aa] Object does not exist on the server: [404] Object does not exist on the server
error: failed to fetch some objects from 'https://github.com/ZekunYangCalvin/ReEDS.git/info/lfs'

[ZekunYangCalvin]

this is my errors after I run git lfs fetch

[ZekunYangCalvin]

I already solved it after redownload it! Thanks!