diff --git a/CHANGELOG.md b/CHANGELOG.md index f4a8c96..1342991 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,12 +7,109 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- Add versioning to the convention +- Mention the adherence to Python convention in README.md + +### Changed + +- Move wiki doc to repo for proper review / versioning +- Disable pyright bytes type promotions + +## [0.2.0] - 2024-02-05 + +### Changed + +- Adopt hatch-openzim for metadata computation +- Upgrade dependencies +- Upgrade to Python 3.11 + 3.12 (instead of 3.10 + 3.11) +- Fix ruff configuration in pyproject.toml to fix depreciation warning in 0.2 +- Fix Pyright ignore rules to adapt to breaking changes in 1.1.348 + +## [0.1.10] - 2023-12-12 + +### Fixed + +- Add package location to adapt to hatchling 1.19.0 + +## [0.1.9] - 2023-09-19 + +### Changed + +- Split Docker build for efficiency + +## [0.1.8] - 2023-08-17 + +### Added + +- Add sample Docker test configuration for daemon processes +- Add support for coverage HTML report + ### Fixed -- Dependency installation issue (#2) +- Fix few invoke task arguments and help + + +## [0.1.7] - 2023-08-04 + +### Changed + +- Update dependencies + +## [0.1.6] - 2023-07-26 + +### Changed + +- Enhance CI to do more tests regarding Docker and Python build and publish dev image + +## [0.1.5] - 2023-07-24 + +### Changed + +- Source Python version from pyproject.toml +- Build Docker image in CI `Publish.yaml` + +### Fixed + +- Adjust Ruff rules ignored by default +- Use `no-cache-dir` for package install + +## [0.1.4] - 2023-07-17 + +### Added + +- Added debugpy + +## [0.1.3] - 2023-07-14 + +### Added + +- Added a `check` feature in hatch +- Added pyright wrapper to this feature +- Installed this feature in `dev` environment +- Used `check-pyright` task in QA workflow + +### Changed + +- Use major versions for workflows actions +- Enable `dev` in default hatch environment + +## [0.1.2] - 2023-07-13 + +### Fixed + +- Fix version to comply with SemVer + +## [0.1.1] - 2023-07-11 + +### Added + +- Add the scripts to lint's features, otherwise we can use the hatch run lint:* + -## [0.1] - 2023-01-01 +## [0.1.0] - 2023-06-22 ### Added -- Ability to do stuff +- Initial version diff --git a/README.md b/README.md index 7c387d2..709b52b 100644 --- a/README.md +++ b/README.md @@ -31,4 +31,6 @@ You'd want to install it in a dedicated virtual-environment (`python3 -m venv so ❯ pip install -e . ``` -Great-project adheres to openZIM's [Contribution Guidelines](https://github.com/openzim/overview/wiki/Contributing) +Great-project adheres to openZIM's [Contribution Guidelines](https://github.com/openzim/overview/wiki/Contributing). + +Great-project has implemented openZIM's [Python bootstrap, conventions and policies](https://github.com/openzim/_python-bootstrap/docs/Policy.md) **v0.2.0**. diff --git a/docs/.gitignore.md b/docs/.gitignore.md new file mode 100644 index 0000000..de695ac --- /dev/null +++ b/docs/.gitignore.md @@ -0,0 +1,5 @@ +Git ignore is built from [gitignore.io](https://www.toptal.com/developers/gitignore) so it's reproducible. We use the `Python` and `macOS` recipes at least. + +Should there be additional changes, those are included at the beginning of the file, before the gitignore.io comment. A comment explains the purpose of the change. + +Should a change be necessary at the bottom of the file, it is mentioned at the beginning of the file. \ No newline at end of file diff --git a/docs/.pre-commit-config.yaml.md b/docs/.pre-commit-config.yaml.md new file mode 100644 index 0000000..ba506c8 --- /dev/null +++ b/docs/.pre-commit-config.yaml.md @@ -0,0 +1,6 @@ +[pre-commit](https://pre-commit.com/) shared configuration to help ensure commits are linted properly. + +Although ruff handles trailing white spaces and EOF checking and fixing, those are very common mistakes which doesn't require much thinking. +It's thus more convenient to keep them than inspect ruff's output and select only appropriate rules to launch with `--fix`. + +**Note**: `pyright` precommit-hook uses system (in `PATH`) `pyright` binary. `pyright` is a node cli tool that must be installed separately. \ No newline at end of file diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md new file mode 100644 index 0000000..ca0a93c --- /dev/null +++ b/docs/CHANGELOG.md @@ -0,0 +1,3 @@ +Mandatory for a versioned project. Must follow [Keep A Changelog](https://keepachangelog.com/en/1.1.0/)'s format. + +Ideally, entries in response to Github issues should mention them. \ No newline at end of file diff --git a/docs/Developer-Setup.md b/docs/Developer-Setup.md new file mode 100644 index 0000000..bdaafdf --- /dev/null +++ b/docs/Developer-Setup.md @@ -0,0 +1,95 @@ +Many different setups are possible. There is no _right_ way of setting up your environment. The following serves as self-documentation and example. + +## [rgaudin](https://github.com/rgaudin)'s with Sublime Text + +- macOS +- Python for each minor stable using [python.org packages](https://www.python.org/downloads/) +- [hatch](https://pypi.org/project/hatch/) installed globally +- [Sublime Text 4](https://www.sublimetext.com/) (proprietary) + - [LSP](https://lsp.sublimetext.io/) + - [LSP-ruff](https://packagecontrol.io/packages/LSP-ruff) + - [LSP-pyright](https://github.com/sublimelsp/LSP-pyright) + - [Debugger](https://github.com/daveleroy/SublimeDebugger) + - [sublack](https://github.com/jgirardet/sublack) + - [TOML](https://github.com/jasonwilliams/sublime_toml_highlighting) + - [Trailing Spaces](https://github.com/SublimeText/TrailingSpaces) + - _non-required but useful for openZIM projects_ + - [Vue Syntax Highlighting](https://github.com/vuejs/vue-syntax-highlight) + - [Pretty YAML](https://github.com/aukaost/SublimePrettyYAML) + - [LSP-json](https://github.com/sublimelsp/LSP-json) + - [LSP-dockerfile](https://github.com/sublimelsp/LSP-dockerfile) + - [JSON Reindent](https://github.com/ThomasKliszowski/json_reindent) + - [Dockerfile Syntax Highlighting](https://github.com/asbjornenge/Docker.tmbundle) + +```sh +# create (if missing) default env, installing dependencies +# also verifies default env's python version (use a matrix for default with a single value to set otherwise) +hatch run python -V + +# make sure it's correct +hatch env show +hatch env find + +# symlink hatch's default environment into `./.venv` (in project) +ln -s "$(hatch env find)" .venv +``` + +The key part is the symlink to `.venv` so that LSP-pyright and other tool can automatically find the environment and properly detect the installed dependencies. + + +## [benoit74](https://github.com/benoit74)'s with Visual Studio Code + +- macOS +- [brew](https://brew.sh/) +- [pyenv](https://github.com/pyenv/pyenv) (installed with brew) +- Python: + - at least each supported and released minor versions (i.e. security + bugfix on https://devguide.python.org/versions/) + - installed with `pyenv install x.y.z` + - all supported and released minor versions are set in `/Users//.python-version` (i.e. available when looking for a version with python, python3, python3.x, ...) +- [hatch](https://pypi.org/project/hatch/) installed globally +- [Visual Studio Code](https://github.com/microsoft/vscode) with following extensions (more or less related to Python development) + - Black Formater (Microsoft) + - Dev Containers (Microsoft) + - Docker (Microsoft) + - Excalidraw (pomdtr) + - Even Better TOML (tamasfe) + - GitLens (GitKraken) + - Jupyter (Microsoft) + - Pylance (Microsoft) + - Python (Microsoft) + - Remote Development (Microsoft) + - reStructuredText (LeXtudio Inc.) + - Ruff (Astral Software) + - Volar (Vue) + - YAML (Red Hat) + +Hatch is configured with an additional section in `config.toml` (configuration file is located in `~/Library/Application Support/hatch`, see https://hatch.pypa.io/latest/config/hatch/, correct file is shown by `hatch status` in any case). +``` +[dirs.env] +virtual = ".hatch" +``` + +This additional Hatch config section ensures that all virtual environments (the only built-in environment in hatch is `virtual` - for now at least) are created in a `.hatch` subfolder inside the project folder ; it is mandatory so that Visual Studio Code will be able to find these dependencies. + +On every project, create a local `.vscode/settings.json` to automatically format your code (adjust `typeCheckingMode` depending on your project): +``` json +{ + + "[python]": { + "editor.defaultFormatter": "ms-python.black-formatter", + "editor.formatOnSave": true, + "editor.codeActionsOnSave": { + "source.organizeImports": true + }, + }, + "python.analysis.typeCheckingMode": "basic", +} +``` + +When starting to work on a project, start a shell on default env + installing dependencies + start VSCode ; starting VSCode from inside the hatch shell ensures he will find automatically the proper Python version and dependencies + +```sh +hatch shell +pip install '.[dev]' +code . +``` \ No newline at end of file diff --git a/docs/Dockerfile-and-Publish-workflow.md b/docs/Dockerfile-and-Publish-workflow.md new file mode 100644 index 0000000..1516eaa --- /dev/null +++ b/docs/Dockerfile-and-Publish-workflow.md @@ -0,0 +1,64 @@ +You will find in this page our guidelines to: +- create a `Dockerfile` to build a Docker image +- create a Github workflow to build and publish the Docker image to a registry + +# Dockerfile + +## Base image + +- Debian-based images using the current [debian stable](https://www.debian.org/releases/) release (e.g. bookworm as of July 2023) +- `slim` variant whenever it is sufficient +- `alpine` variant only for highly constrained space environment (Offspot apps for instance). See [drawbacks](https://github.com/tiangolo/uvicorn-gunicorn-docker#-alpine-python-warning) for explanations +- October 2023, latest Python stable is `3.12` and Debian stable is `bookworm` hence `python:3.12-slim-bookworm` or `python:3.12-bookworm` + +## RUN commands + +- when using multiple lines, add a space before the `\` sign +- when linking multiple command, place the `&&` sign at the beginning of the next line +- on multiple lines, align the start of the commands, not the `&&` +- when listing packages to be installed by apt, add each of the on a new line, one per line, alphabetically order +- do not hesitate to add comments between lines to explain what is the reason / purpose of the next line +- do not include `apt-get clean` command since official Debian and Ubuntu images [automatically run apt-get clean](https://github.com/moby/moby/blob/03e2923e42446dbb830c654d0eec323a0b4ef02a/contrib/mkimage/debootstrap#L82-L105), so explicit invocation is not required + + +Sample: + +```dockerfile + +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + # locales required if tool has any i18n support + locales-all \ + && rm -rf /var/lib/apt/lists/* \ + && python -m pip install --no-cache-dir -U \ + build \ + pip +``` + +## CMD + +- add a command which displays the program's help + +# Github Workflow + +## When + +Publishing to the registry should be done on Github release event. + +``` +on: + release: + types: [published] +``` + +## Publish action + +Except if not appropriate, we recommend to use our specific Github action : [openzim/docker-publish-action](https://github.com/openzim/docker-publish-action) which handles building the image and publishing it. + +Some quirks not to forget: + +- Image name convention is to use dashes not underscores + +## Registry + +We publish our images on ghcr.io \ No newline at end of file diff --git a/docs/Github-Workflows.md b/docs/Github-Workflows.md new file mode 100644 index 0000000..9cba378 --- /dev/null +++ b/docs/Github-Workflows.md @@ -0,0 +1,3 @@ +- `QA.yml` runs linting and static type checks (always) +- `Tests.yml` runs the tests on all supported Python versions. On latest version, uploads coverage to codecov. Also builds the package (to ensure build is not broken) +- `Publish.yaml` builds the packages and uploads them to PyPI \ No newline at end of file diff --git a/docs/LICENSE.md b/docs/LICENSE.md new file mode 100644 index 0000000..796f6b7 --- /dev/null +++ b/docs/LICENSE.md @@ -0,0 +1,5 @@ +Should be the _GNU General Public License v3.0_. + +We use the Github's license feature to add it (raw, complete, text). + +See: [`GPL-3.0`](https://github.com/github/choosealicense.com/blob/gh-pages/_licenses/gpl-3.0.txt) \ No newline at end of file diff --git a/docs/Policy.md b/docs/Policy.md new file mode 100644 index 0000000..2d6b2d4 --- /dev/null +++ b/docs/Policy.md @@ -0,0 +1,27 @@ +This openZIM Python Bootstrap setup (`main` branch) represents the **current** recommendations for setting-up a Python project. + +- It should be used as base for any new Python project +- It should serve as a reference to migrate older Python projects +- Requirements for older projects are relaxed: + - Static Type Checking: + - `pyright` mode might be set to `basic` (with the `typeCheckingMode` parameter) + - few `# pyright: ignore` comments are allowed in the code + - Ruff rules: + - `# noqa: xxx` comments can be placed when necessary + - as a last resort (because it means that the situation might continue to get worse), some rule or groups might be completely commented in `pyproject.toml` + - Cython-using projects should continue with setuptools/setup.py + +## Contributing + +Discussions on significant improvements should happen via [Opening an Issue](https://github.com/openzim/_python-bootstrap/issues/new). + +Simpler suggestions are expected as [Pull Request](https://github.com/openzim/_python-bootstrap/compare) + + +## Versioning + +The Python policy / bootstrap / convention is versioned in the main CHANGELOG.md. + +Tags are used to point to the various versions. + +Project must specify in their README.md the version they currently support to track required changes. \ No newline at end of file diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..f6c588f --- /dev/null +++ b/docs/README.md @@ -0,0 +1,13 @@ +README (markdown format) should at least contain (in this order): + +- project name +- single line, general, description +- collection of badges: + - [Codefactor](https://www.codefactor.io) + - License + - Code coverage via codecov.io + - [PyPi](https://pypi.org) + - Python Version +- Installation/usage instructions +- Contribution guidelines (or at least a link to [Contributing](https://github.com/openzim/overview/wiki/Contributing)) +- A reference to the fact the project adheres to this boostrap conventions and current conventions version in place \ No newline at end of file diff --git a/docs/Usage.md b/docs/Usage.md new file mode 100644 index 0000000..d0bc67c --- /dev/null +++ b/docs/Usage.md @@ -0,0 +1,51 @@ +If you've not already read it, please check our [Policy](./Policy.md) first. + +## Using Hatch + +```sh +❯ pip install hatch + +# local install (in default env) / re-sync packages +❯ hatch run pip list +❯ pre-commit install + +# scripts discovery +❯ hatch env show + +# linting, testing, coverage, checking +❯ hatch run lint:all +❯ hatch run lint:fixall +# run tests on all matrixed' envs +❯ hatch run test:run +# run tests in a single matrixed' env +❯ hatch env run -e test -i py=3.11 coverage +# run static type checks +❯ hatch env run check:all + +# building packages +❯ hatch build +``` + +## _Bare_ Python + +```sh +❯ python3 -m venv .env && source .env/bin/activate + +# local install / install newly added packages +❯ pip install -e .[dev] +❯ pre-commit install + +# scripts discovery +❯ inv -l + +# linting, testing, coverage, static type checks +❯ inv lintall +❯ inv fixall +❯ inv test +❯ inv coverage +❯ inv checkall + +# building packages +❯ pip install build +❯ python3 -m build +``` \ No newline at end of file diff --git a/docs/Versioning.md b/docs/Versioning.md new file mode 100644 index 0000000..456c3d1 --- /dev/null +++ b/docs/Versioning.md @@ -0,0 +1 @@ +We use [semantic versioning](https://semver.org/) for published software. _Internal tools_ (Web projects) are deployed off the main branch and are version-less. diff --git a/docs/__about__.py.md b/docs/__about__.py.md new file mode 100644 index 0000000..f38069e --- /dev/null +++ b/docs/__about__.py.md @@ -0,0 +1 @@ +Contains `__version__` string at least. \ No newline at end of file diff --git a/docs/pyproject.toml.md b/docs/pyproject.toml.md new file mode 100644 index 0000000..41b1a82 --- /dev/null +++ b/docs/pyproject.toml.md @@ -0,0 +1,45 @@ +Python setup uses `hatchling` as build backend and leans on [`hatch`](https://hatch.pypa.io) for various features. + +### Target Python Version + +Target depends on the audience of the project: +- Specific version if audience has specific requirement (eg. raspberry-pi) +- Latest stable version for internal tools (eg. docker deployed) +- [Latest still maintained version](https://en.wikipedia.org/wiki/History_of_Python#Table_of_versions) for larger audience (eg. library) + +### Version + +Version must be dynamic and set once in code. `module.__about__` usually. + +### Dependencies + +- Solely specified in this file. +- Projects that are not meant to interact with others (tools vs library) use strict matching versions. +- Libraries and alike uses ranges when possible (semantic versioning). +- Only direct dependencies (unless required to bypass a sub-dependency issue) + +### Linting + +- Black (latest available version) +- Ruff for import sorting (isort-like) and many other checks +- Dedicated hatch environment +- Black and ruff configuration so those works without hatch as well +- Scripts for just black and just ruff checking +- Scripts for black fixing and ruf fixing (accepts params) +- Scripts for global checking and fixing +- Sample ruff configuration + +### Checking + +- [pyright](https://microsoft.github.io/pyright) for static type checking +- Dedicated hatch environment +- Sample pyright configuration +- Scripts for running pyright + +### QA + +- Tests and coverage reporting with [`pytest`](https://docs.pytest.org) +- Dedicated hatch environment +- Hatch environment matrix to easily tests on multiple python versions +- pytest and coverage configuration so those works without hatch as well +- Scripts for running tests (accepts params), running with coverage and reporting \ No newline at end of file diff --git a/docs/tasks.py.md b/docs/tasks.py.md new file mode 100644 index 0000000..506ad14 --- /dev/null +++ b/docs/tasks.py.md @@ -0,0 +1 @@ +[Invoke](https://www.pyinvoke.org/) Commands for linting and QA. \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index 9004a3e..3001413 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -219,3 +219,4 @@ exclude = [".env/**", ".venv/**"] extraPaths = ["src"] pythonVersion = "3.11" typeCheckingMode="strict" +disableBytesTypePromotions = true