Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[f]: Update dev, CI and packaging config #23

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Conversation

SimmonsRitchie
Copy link

@SimmonsRitchie SimmonsRitchie commented Jun 18, 2024

What's this PR do?

Updates the packaging configuration, CI/CD process, and the dev environment of this project. Changes include:

  • setup.py removed and pyproject.toml updated with package metadata.
  • CI and deploy github actions updated.
  • requirements.txt replaced with pipenv.
  • Minimum Python version bumped from 3.6 to 3.8
  • PR template added.

Why are we doing this?

These changes are intended to make local dev and packaging a bit smoother and more modern.

Packaging
Replacing setup.py with pyproject.toml is intended to provide the following advantages:

  • Standardization: Aligns with modern Python packaging standards (PEP 517 and PEP 518).
  • Clarity: Provides a clearer and more structured way to specify package metadata and dependencies.
  • Tooling Support: Enhanced support for modern packaging and build tools.
  • Future-proofing: Prepares the project for future improvements in Python's packaging ecosystem.

Pipenv
We believe pipenv makes it easier to manage dependencies in local dev, particularly dev dependencies, than the use of requirements.txt. The use of pipenv is also consistent with our other city-scraper projects.

Python 3.8
All or nearly all of the city-scraper projects now require Python 3.8 or higher. Requiring 3.8 or higher in this package is intended to better future-proof the city-scraper project and avoid unexpected dependency issues.

Steps to manually test

The distribution code for this PR has been uploaded to TestPyPi for testing and can be found here. To test that it's installing and working correctly:

  1. Open an existing city-scrapers project locally or clone a city-scrapers project:

git clone https://github.com/City-Bureau/city-scrapers-wichita.git

  1. If you have previously worked with the project, remove the existing virtual env:

pipenv --rm

  1. In the project, update the Pipfile to include TestPyPi as a source. You will now have two sources:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
name = "testpypi"
url = "https://test.pypi.org/simple"
verify_ssl = true
  1. Further update the Pipfile so that the reference to city-scrapers-core targets the TestPyPi distribution:
city-scrapers-core = {version="0.11.0", extras=["azure"], index="testpypi"}
  1. Install the project and shell into the pipenv virtual environment:

pipenv sync && pipenv shell

  1. Run one of the scrapers in the repo. For instance, in city-scrapers-wichita,

scrapy crawl wicks_boe -O wicks_boe.csv

You should be able to successfully install the distribution package and run a spider successfully.

Are there any smells or added technical debt to note?

  • The deploy Github action, which builds the dist code and uploads the package to PyPi, has not been tested. The action was slightly altered in this PR. Based on the current configuration of this action, it will be triggered when a tag is pushed to the repository that matches the pattern "v*". I intended to do this after this PR is merged into master. It's possible the action could fail and a subsequent PR might be needed to fix this action. Of particular note, this action makes use of the create-release github action, which appears to have been archived more than three years ago (Mar 3, 2021), and could cause unexpected behavior.

@SimmonsRitchie SimmonsRitchie changed the title Tweak python env in CI [f]: Modernize packaging and update CI Jun 20, 2024
Migrate packaging to pyproject.toml

- Removes setup.py file.
- Updates pyproject.toml file to include package metadata and deps.
- Updates CI/CD config
- Requires python 3.8 or higher
@SimmonsRitchie SimmonsRitchie changed the title [f]: Modernize packaging and update CI [f]: Modernize dev and packaging config Jun 24, 2024
@SimmonsRitchie SimmonsRitchie changed the title [f]: Modernize dev and packaging config [f]: Modernize dev, CI and packaging config Jun 24, 2024
@SimmonsRitchie SimmonsRitchie changed the title [f]: Modernize dev, CI and packaging config [f]: Update dev, CI and packaging config Jun 24, 2024
@SimmonsRitchie SimmonsRitchie marked this pull request as ready for review June 24, 2024 17:44
Copy link

@LienDang LienDang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants