Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regenerate pipenv lockfiles used as test fixtures #7697

Closed
jeffwidman opened this issue Aug 2, 2023 · 2 comments
Closed

Regenerate pipenv lockfiles used as test fixtures #7697

jeffwidman opened this issue Aug 2, 2023 · 2 comments
Assignees
@jeffwidman
Labels
Ecosystems Used by the maintainer team for internal-facing project tracking L: python:pipenv Python packages via pipenv T: tech-debt ⚙️

Comments

@jeffwidman
Copy link
Member

jeffwidman commented Aug 2, 2023 •
edited
Loading

Code improvement description

Need to update/regenerate the pipenv lockfiles used as test fixtures:
https://github.com/dependabot/dependabot-core/tree/4b9fd12bf441c45038dabeaabe8011d6ab04a1bc/python/spec/fixtures/lockfiles

I started doing this because some of the lockfiles cause test failures with:

However, I realized that because we currently pin to an older version of pipenv:

dependabot-core/python/helpers/requirements.txt

Line 4 in 4b9fd12

pipenv==2022.4.8

We'll need to do this again once we update pipenv... so IMO no point in doing this just yet.

Related:
@jeffwidman jeffwidman added T: tech-debt ⚙️ L: python:pipenv Python packages via pipenv Ecosystems Used by the maintainer team for internal-facing project tracking labels Aug 2, 2023
jeffwidman added a commit that referenced this issue Aug 2, 2023
@jeffwidman
Delete deprecated host-environment-markers key
75436d9 
Our `pipenv` fixtures are a bit outdated... when I tried to delete
Python 3.6, they started failing because we've got some code that reads
the python version and tries to use that to guess at what version of
python to run.

However, when I tried regenerating one of these lockfiles under a newer
Python, I was surprised that the entire `host-environment-markers` key
disappeared.

After a little digging, I found that this key was removed in 2018, which
also explains why only the older fixtures have it and not the more
recent fixtures. More historical context:
* pypa/pipenv#753
* pypa/pipenv@ecf7842#r28870534
* pypa/pipenv@ecf7842#diff-ef852c4ac364f946819f765a6bc26f04f1b0968f31fc512949a60fa2ab0685e8L1201

In order to update our lockfiles, I first started by running
`PYENV_VERSION=3.11.4 pyenv exec pipenv lock`... However, I ran into
some problems because:
1. Some of the lockfiles are simply unresolvable by design in order to
   stress test our code. So they have to be manually munged around.
2. We are currently pinned to an olde version of `pipenv` due to
   conflicts with Python `3.6`, etc: #6104

Since we will soon be dropping Python 3.6, 3.7, and then planning to
update `pipenv` version, I decided it would be more efficient from an
engineering perspective to hack my way through the jungle with a machete
for a little bit by:
1. Manually delete the key from the lockfiles (that's what this PR does)
2. File a ticket to track regenerating the `pipenv` lockfiles: #7697
3. Finish the work of deprecating EOL'd python versions and updating
   `pipenv`
4. Then finally regenerate the fixtures using `pipenv lock`.

So in this PR I manually deleted the key... but note that I did so
manually not programmatically, so these fixtures may still be outdated
in other ways.
@jeffwidman jeffwidman mentioned this issue Aug 2, 2023
Merged
jeffwidman added a commit that referenced this issue Aug 2, 2023
@jeffwidman
Delete deprecated host-environment-markers key (#7698)
1bbea43 
Our `pipenv` fixtures are a bit outdated... when I tried to delete
Python 3.6, they started failing because we've got some code that reads
the python version and tries to use that to guess at what version of
python to run.

However, when I tried regenerating one of these lockfiles under a newer
Python, I was surprised that the entire `host-environment-markers` key
disappeared.

After a little digging, I found that this key was removed in 2018, which
also explains why only the older fixtures have it and not the more
recent fixtures. More historical context:
* pypa/pipenv#753
* pypa/pipenv@ecf7842#r28870534
* pypa/pipenv@ecf7842#diff-ef852c4ac364f946819f765a6bc26f04f1b0968f31fc512949a60fa2ab0685e8L1201

In order to update our lockfiles, I first started by running
`PYENV_VERSION=3.11.4 pyenv exec pipenv lock`... However, I ran into
some problems because:
1. Some of the lockfiles are simply unresolvable by design in order to
   stress test our code. So they have to be manually munged around.
2. We are currently pinned to an olde version of `pipenv` due to
   conflicts with Python `3.6`, etc: #6104

Since we will soon be dropping Python 3.6, 3.7, and then planning to
update `pipenv` version, I decided it would be more efficient from an
engineering perspective to hack my way through the jungle with a machete
for a little bit by:
1. Manually delete the key from the lockfiles (that's what this PR does)
2. File a ticket to track regenerating the `pipenv` lockfiles: #7697
3. Finish the work of deprecating EOL'd python versions and updating
   `pipenv`
4. Then finally regenerate the fixtures using `pipenv lock`.

So in this PR I manually deleted the key... but note that I did so
manually not programmatically, so these fixtures may still be outdated
in other ways.
jeffwidman added a commit that referenced this issue Aug 2, 2023
@jeffwidman
Move the Pipfile/Pipfile.lock fixtures to a clearly named folder
46ee98b 
This does two things:
1. Moves the files in the `lockfiles` directory to the `pipfile_files` directory.
2. Moves the files in the `pipfiles` directory to the `pipfile_files`
   directory.

All the files in `lockfiles` are actually specific to `pipenv` and their
`pipfile.lock` file format. So I was a bit surprised because `poetry`
also has a lockfile format... but those are over in `pyproject_locks`
dir.

I think what happened is that when `pipenv` popularized the
`Pipfile`/`Pipfile.lock` format, there was a lot of initial excitement
that it might become _the_ lockfile format within Python. So Grey may
have called it `lockfile` assuming that other tooling would start to
migrate towards this. However, `pipenv` made some design choices that
made it a ["fails-to-take-over-the-world-because-it-tries-to-do-too-much"](https://hynek.me/articles/python-app-deps-2018/) type of
tool.

At first I started by merely renaming the folder from `lockfiles` to
`pipfile_lockfiles`.

However, while I was working regenerating the `pipfile.lock` files
(#7697) I found it a
royal pain to jump back and forth between two folders where one holds the
input `Pipfile` and the other holds the output `Pipfile.lock` files. And
the lockfiles already have a `.lock` suffix. So instead merging them
into a single folder makes it easier to spot when a file should be
regenerated into a lockfile vs when it'll fail to even generate a
lockfile.

So this hopefully makes things a lot clearer / less ambiguous for anyone
working on this section of code.
@jeffwidman jeffwidman mentioned this issue Aug 2, 2023
Merged
jeffwidman added a commit that referenced this issue Aug 2, 2023
@jeffwidman
Move the Pipfile/Pipfile.lock fixtures to a clearly named folder (#…
6edb248 
…7700)

This does two things:
1. Moves the files in the `lockfiles` directory to the `pipfile_files` directory.
2. Moves the files in the `pipfiles` directory to the `pipfile_files`
   directory.

All the files in `lockfiles` are actually specific to `pipenv` and their
`pipfile.lock` file format. So I was a bit surprised because `poetry`
also has a lockfile format... but those are over in `pyproject_locks`
dir.

I think what happened is that when `pipenv` popularized the
`Pipfile`/`Pipfile.lock` format, there was a lot of initial excitement
that it might become _the_ lockfile format within Python. So Grey may
have called it `lockfile` assuming that other tooling would start to
migrate towards this. However, `pipenv` made some design choices that
made it a ["fails-to-take-over-the-world-because-it-tries-to-do-too-much"](https://hynek.me/articles/python-app-deps-2018/) type of
tool.

At first I started by merely renaming the folder from `lockfiles` to
`pipfile_lockfiles`.

However, while I was working regenerating the `pipfile.lock` files
(#7697) I found it a
royal pain to jump back and forth between two folders where one holds the
input `Pipfile` and the other holds the output `Pipfile.lock` files. And
the lockfiles already have a `.lock` suffix. So instead merging them
into a single folder makes it easier to spot when a file should be
regenerated into a lockfile vs when it'll fail to even generate a
lockfile.

So this hopefully makes things a lot clearer / less ambiguous for anyone
working on this section of code.
@jeffwidman jeffwidman mentioned this issue Aug 3, 2023
Closed
@jeffwidman jeffwidman self-assigned this Aug 3, 2023
@jeffwidman jeffwidman mentioned this issue Aug 10, 2023
Merged
@deivid-rodriguez
Copy link
Contributor

Why is this needed? We just updated pipenv but I don't think we dropped support for any lockfile format. So I guess it's still useful to keep testing against previous lockfiles? I guess if there's been some change in the format we should also add tests with it, but no sure if there's a clear benefit on explicitly migrating all lockfiles now?

brettfo pushed a commit to brettfo/dependabot-core that referenced this issue Oct 11, 2023
@jeffwidman
Delete deprecated host-environment-markers key (dependabot#7698)
a2a8900 
Our `pipenv` fixtures are a bit outdated... when I tried to delete
Python 3.6, they started failing because we've got some code that reads
the python version and tries to use that to guess at what version of
python to run.

However, when I tried regenerating one of these lockfiles under a newer
Python, I was surprised that the entire `host-environment-markers` key
disappeared.

After a little digging, I found that this key was removed in 2018, which
also explains why only the older fixtures have it and not the more
recent fixtures. More historical context:
* pypa/pipenv#753
* pypa/pipenv@ecf7842#r28870534
* pypa/pipenv@ecf7842#diff-ef852c4ac364f946819f765a6bc26f04f1b0968f31fc512949a60fa2ab0685e8L1201

In order to update our lockfiles, I first started by running
`PYENV_VERSION=3.11.4 pyenv exec pipenv lock`... However, I ran into
some problems because:
1. Some of the lockfiles are simply unresolvable by design in order to
   stress test our code. So they have to be manually munged around.
2. We are currently pinned to an olde version of `pipenv` due to
   conflicts with Python `3.6`, etc: dependabot#6104

Since we will soon be dropping Python 3.6, 3.7, and then planning to
update `pipenv` version, I decided it would be more efficient from an
engineering perspective to hack my way through the jungle with a machete
for a little bit by:
1. Manually delete the key from the lockfiles (that's what this PR does)
2. File a ticket to track regenerating the `pipenv` lockfiles: dependabot#7697
3. Finish the work of deprecating EOL'd python versions and updating
   `pipenv`
4. Then finally regenerate the fixtures using `pipenv lock`.

So in this PR I manually deleted the key... but note that I did so
manually not programmatically, so these fixtures may still be outdated
in other ways.
brettfo pushed a commit to brettfo/dependabot-core that referenced this issue Oct 11, 2023
@jeffwidman
Move the Pipfile/Pipfile.lock fixtures to a clearly named folder (d…
b21fe8f 
…ependabot#7700)

This does two things:
1. Moves the files in the `lockfiles` directory to the `pipfile_files` directory.
2. Moves the files in the `pipfiles` directory to the `pipfile_files`
   directory.

All the files in `lockfiles` are actually specific to `pipenv` and their
`pipfile.lock` file format. So I was a bit surprised because `poetry`
also has a lockfile format... but those are over in `pyproject_locks`
dir.

I think what happened is that when `pipenv` popularized the
`Pipfile`/`Pipfile.lock` format, there was a lot of initial excitement
that it might become _the_ lockfile format within Python. So Grey may
have called it `lockfile` assuming that other tooling would start to
migrate towards this. However, `pipenv` made some design choices that
made it a ["fails-to-take-over-the-world-because-it-tries-to-do-too-much"](https://hynek.me/articles/python-app-deps-2018/) type of
tool.

At first I started by merely renaming the folder from `lockfiles` to
`pipfile_lockfiles`.

However, while I was working regenerating the `pipfile.lock` files
(dependabot#7697) I found it a
royal pain to jump back and forth between two folders where one holds the
input `Pipfile` and the other holds the output `Pipfile.lock` files. And
the lockfiles already have a `.lock` suffix. So instead merging them
into a single folder makes it easier to spot when a file should be
regenerated into a lockfile vs when it'll fail to even generate a
lockfile.

So this hopefully makes things a lot clearer / less ambiguous for anyone
working on this section of code.
@deivid-rodriguez
Copy link
Contributor

Closing since there was no more feedback.

@deivid-rodriguez deivid-rodriguez closed this as not planned Won't fix, can't repro, duplicate, stale Nov 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jeffwidman jeffwidman

Labels
Ecosystems Used by the maintainer team for internal-facing project tracking L: python:pipenv Python packages via pipenv T: tech-debt ⚙️
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jeffwidman @deivid-rodriguez