[f]: Update dev, CI and packaging config #23
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's this PR do?
Updates the packaging configuration, CI/CD process, and the dev environment of this project. Changes include:
setup.py
removed andpyproject.toml
updated with package metadata.requirements.txt
replaced withpipenv
.Why are we doing this?
These changes are intended to make local dev and packaging a bit smoother and more modern.
Packaging
Replacing
setup.py
withpyproject.toml
is intended to provide the following advantages:Pipenv
We believe
pipenv
makes it easier to manage dependencies in local dev, particularly dev dependencies, than the use ofrequirements.txt
. The use ofpipenv
is also consistent with our other city-scraper projects.Python 3.8
All or nearly all of the
city-scraper
projects now require Python 3.8 or higher. Requiring3.8
or higher in this package is intended to better future-proof thecity-scraper
project and avoid unexpected dependency issues.Steps to manually test
The distribution code for this PR has been uploaded to TestPyPi for testing and can be found here. To test that it's installing and working correctly:
git clone https://github.com/City-Bureau/city-scrapers-wichita.git
pipenv --rm
Pipfile
to include TestPyPi as a source. You will now have two sources:Pipfile
so that the reference to city-scrapers-core targets the TestPyPi distribution:pipenv
virtual environment:pipenv sync && pipenv shell
city-scrapers-wichita
,scrapy crawl wicks_boe -O wicks_boe.csv
You should be able to successfully install the distribution package and run a spider successfully.
Are there any smells or added technical debt to note?
deploy
Github action, which builds the dist code and uploads the package toPyPi
, has not been tested. The action was slightly altered in this PR. Based on the current configuration of this action, it will be triggered when a tag is pushed to the repository that matches the pattern "v*". I intended to do this after this PR is merged into master. It's possible the action could fail and a subsequent PR might be needed to fix this action. Of particular note, this action makes use of the create-release github action, which appears to have been archived more than three years ago (Mar 3, 2021), and could cause unexpected behavior.