Skip to content

make_SAM_files() overwrites the same file due to no unique gid identifier for max_workers greater than 1 #222

@juflorez

Description

@juflorez

rex version checks

  • I have confirmed this bug has not already been reported.

  • I have confirmed this bug exists on the latest version of rex.

  • I have confirmed this bug exists on the main branch of rex.

Bug Description

When running make_SAM_files, with max_workers set to 2 or greater, the same file is being overwritten as no gid info is appended to the name. In the parallel process, it breaks up the gids into default chunks of 36 which is great, however in the make_SAM_gid function, if less then n_chunks it is overwriting the filename as there is only 1 gid. I'm assuming you know more about why this filter exists, but either setting a filter if it's a parallel process or just defaulting the name to include the gid, would be fantastic! The specific bug code is in resource_extraction/resource_extraction.py lines 936-938. This is true for both NSRDBX and WindX.

Reproducible Example

from rex import NSRDBX

hsds_data_path = "/nrel/nsrdb/GOES/aggregated/v4.0.0/nsrdb_2023.h5"
gids = [483627,369433,484625]
NSRDBX.make_SAM_files(res_h5=hsds_data_path,gids=gids,out_path="testpath.csv",max_workers=2)

Full Traceback

No response

Expected Behavior

to create unique files when running make_SAM_files with the gid appended, not overwriting the same file with max_workers greater than or equal to 2.

System and installed versions

  • OS: [e.g. OSX, Windows, Linux]
  • Compute hardware: [e.g. Kestrel, AWS]
  • rex Version: [e.g. 0.2.98]
  • Python Version: [e.g. 3.13]
  • Pandas Version: [e.g. 2.1.0]
    ...

Charge code

No response

Metadata

Metadata

Assignees

Labels

p-highPriority: hightopic-resource-handlerIssues/pull requests related to the main rex Resource handler

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions