Skip to content

Incompatible snakemake versions between different environments lead to issues #1489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
2 tasks done
koen-vg opened this issue Jan 10, 2025 · 2 comments · Fixed by #1509
Closed
2 tasks done

Incompatible snakemake versions between different environments lead to issues #1489

koen-vg opened this issue Jan 10, 2025 · 2 comments · Fixed by #1509
Labels
bug
Milestone
v2025.01.0

Comments

@koen-vg
Copy link
Contributor

koen-vg commented Jan 10, 2025

Checklist

  • I am using the current master branch or the latest release. Please indicate.
  • I am running on an up-to-date pypsa-eur environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

When I was just starting a fresh pypsa-eur repo, I was getting the below error during retrieval rules:

AttributeError: 'Snakemake' object has no attribute '_params_store'

After just a little bit of digging, it looks like snakemake has changed how it internally refers to parameters, and my take is that the environment.yaml and retrieve.yaml have different version requirements for snakemake, leading to two different versions, one of which stored parameters in one way, and one in the other way.

The suggested fix is to somehow ensure that exactly the same snakemake version is used in both environments.
@koen-vg koen-vg added the bug label Jan 10, 2025
@lkstrp
Copy link
Member

lkstrp commented Jan 10, 2025

Ref #1469

Hot fix is to simply not use retrieve.yaml and only use environment.yaml/ pinned envs.

The retrieve.yaml was not really maintained and I dropped it in 1469. I think you were the one who added it in the first place. What is the use case for it? I am mainly curious. We can have multiple environments, but they should all have a pinned version and need to be checked/ maintained via the CI. At the moment this is only done for the base environment.yaml.

@koen-vg
Copy link
Contributor Author

koen-vg commented Jan 10, 2025

Yes so if anything this shows that #1469 is well-timed :)

If I remember correctly, one of my main motivations was that I thought it was annoying that development changes in environment.yaml would cause snakemake to trigger reruns for all retrieval rules, since the software environment changed. While we, the developer, might know that bumping the pandas version or whatever doesn't have anything to do with whether the data bundle needs to be retrieved again or not, snakemake doesn't know that.

Granted, there are of course other ways of fixing this, like some sort of snakemake [...] --touch run, or blanket use of --rerun-triggers mtime. If retrieve.yaml causes trouble or is a pain to maintain, then the occasional snakemake --touch run might be worth it.

Hope this makes some sense.

@fneum fneum added this to the v0.14.0 milestone Jan 15, 2025
fneum added a commit that referenced this issue Jan 24, 2025
@fneum
remove retrieve.yaml (closes #1489)
a1ee503
@fneum fneum mentioned this issue Jan 24, 2025
Merged
fneum added a commit that referenced this issue Jan 24, 2025
@fneum @pre-commit-ci
remove retrieve.yaml (#1509)
33664a6 
* remove retrieve.yaml (closes #1489)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
v2025.01.0
Development

Successfully merging a pull request may close this issue.

remove retrieve.yaml PyPSA/pypsa-eur
3 participants
@fneum @lkstrp @koen-vg