From f3a35f394697f23281c16cf257d18683d7646178 Mon Sep 17 00:00:00 2001 From: Jasper Date: Wed, 17 Aug 2022 10:03:06 +0200 Subject: [PATCH] many enhancements to the contributing guide, some cleanups for the Makefile including a new target: `make cleanup` Co-authored-by: Michael R. Crusoe <1330696+mr-c@users.noreply.github.com> --- CONTRIBUTING.md | 78 ++++++++++++++++++++++++++++++++++++------------- Makefile | 39 +++++++++++++------------ 2 files changed, 78 insertions(+), 39 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ad205608c..34526dd50 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,32 +1,68 @@ Style guide: -- PEP-8 -- Python 3.7+ compatible code +- PEP-8 (as implemented by the `black` code formatting tool) +- Python 3.6+ compatible code - PEP-484 type hints -- Vertically align the type hints in function definitions -The development is done using ``git``, we encourage you to get familiar with it. +The development is done using `git`, we encourage you to get familiar with it. Here's a rough guide (improvements are welcome!) To get the code and start working on the changes you can start a console and: -- Clone the cwltool: ``git clone https://github.com/common-workflow-language/cwltool.git`` -- Switch to cwltool directory: ``cd cwltool`` +- Clone the cwltool: `git clone https://github.com/common-workflow-language/cwltool.git` +- Switch to cwltool directory: `cd cwltool` -In order to contribute to the development of ``cwltool``, the source code needs to pass the test before your changes are accepted. -There are a couple ways to test the code with your changes: let `tox` manage installation and test running in virtual environments, or do it manually (preferably in a virtual environment): -- Install ``tox`` preferably using the OS' package manager, otherwise it can be installed with ``pip install --user -U tox`` +In order to contribute to the development of `cwltool`, the source code needs to +pass the test before your changes are accepted. There are a couple ways to test +the code with your changes: let `tox` manage installation and test running in +virtual environments, or do it manually (preferably in a virtual environment): +- Install `tox` preferably using the OS' package manager, otherwise it can be + installed with `pip install --user -U tox` - Make your changes to the code and add tests for new cool things you're adding! -- Run the tests with the command ``tox``, it's recommended to use some parameters as tox will try to run all the checks in all available python interpreters. -- The important tests to run are ``unit tests`` and ``type tests``. - To run these two in Python 3.7, we can tell tox to run only those tests by running: ``tox -e py37-unit,py37-mypy2,py37-mypy3``. -- Run ``tox -l`` to see all available tests and runtimes +- Run the tests with the command `tox`, it's recommended to use some parameters + as tox will try to run all the checks in all available python interpreters. +- The important tests to run are "unit tests" and "type tests". + To run these two in Python 3.7 (for example), we can tell tox to run only those + tests by running: `tox -e py37-mypy`. (the `mypy` tox target also runs the + unit tests; to just run the type checker use `make mypy` as shown below) +- Run `tox -l` to see all available tests and runtimes For the more traditional workflow: -- Create a virtual environment: ``python3 -m venv cwltool`` -- To begin using the virtual environment, it needs to be activated: ``source bin/activate`` -- To check if you have the virtual environment set up: ``which python`` and it should point to python executable in your virtualenv -- Install cwltool: ``pip install -e .`` -- Check the version which might be different from the version installed in general on any system: ``cwltool --version`` +- Create a virtual environment: `python3 -m venv cwltool` + (if you have multiple version of Python installed, then you can replace `python3` + in this command with the version you want to use, `python3.10` for example) +- To begin using the virtual environment, it needs to be activated: `source venv/bin/activate` +- To check if you have the virtual environment set up: `which python` + and it should point to python executable in your virtualenv +- Install cwltool in development mode, along with development tools: + `make install-dep dev` +- Check the `cwltool` version which might be different from the version installed in + general on your system: `cwltool --version` - Make your changes to the code and add tests for new cool things you're adding! -- Run the unit-tests to see : ``python setup.py test`` -- After you're done working on ``cwltool``, you can deactivate the virtual environment: ``deactivate`` +- Run the unit-tests to see : `make test` +- After you're done working on `cwltool`, you can deactivate the virtual + environment: `deactivate` -When tests are passing, you can simply commit and create a PR on ``cwltool`` repo \ No newline at end of file +Before you commit your code (or at least before you push to GitHub) it is +recommended to run `make cleanup` +(which is a shortcut for `make sort_imports format flake8 diff_pydocstyle_report`) +to fix common issues and point out any remaining code formatting issues +that cannot be fixed automatically. + +When tests are passing, you can create git commits in a new branch and push to +GitHub to make a pull request on `cwltool` repo + +Useful `make` commands: +The `make` commands below can help you to install the dependencies, format and test your code + +- `make help` to show the list of commands that can be used with the make function +- `make install-dep` will install the dependencies needed by `cwltool` +- `make format` will clean up your code according to the accepted python standard +- `make test` will perform the tests of `cwltool` (can take a while to run all tests) +- `make sort_imports` will clean up and sort your import statements +- `make remove_unused_imports` will remove any unneeded `import` statements +- `make flake8` will catch format issues that `make format` cannot fix automatically +- `make diff_pydocstyle_report` check Python docstring style for changed files only +- `make diff-cover` to run the tests and point out which lines of code that have been + changed as compared to the `main` branch are missing test coverage. This is also calculated + automatically after all the automatic tests run when you open a Pull Request on GitHub; + running it locally can help you confirm that any changes or additions + to the tests are sufficient before pushing to GitHub. +- `make cleanup` is a shortcut for `make sort_imports format flake8 diff_pydocstyle_report` diff --git a/Makefile b/Makefile index b3bceda59..7b04c1dcf 100644 --- a/Makefile +++ b/Makefile @@ -36,14 +36,17 @@ VERSION=3.1.$(shell TZ=UTC git log --first-parent --max-count=1 \ mkfile_dir := $(dir $(abspath $(lastword $(MAKEFILE_LIST)))) UNAME_S=$(shell uname -s) -## all : default task +## all : default task (install cwltool in dev mode) all: dev -## help : print this help message and exit +## help : print this help message and exit help: Makefile @sed -n 's/^##//p' $< -## install-dep : install most of the development dependencies via pip +## cleanup : shortcut for "make sort_imports format flake8 diff_pydocstyle_report" +cleanup: sort_imports format flake8 diff_pydocstyle_report + +## install-dep : install most of the development dependencies via pip install-dep: install-dependencies install-dependencies: FORCE @@ -53,20 +56,20 @@ install-dependencies: FORCE install-doc-dep: pip install -r docs/requirements.txt -## install-deb-dep: install most of the dev dependencies via apt-get +## install-deb-dep : install many of the dev dependencies via apt-get install-deb-dep: sudo apt-get install $(DEBDEVPKGS) -## install : install the ${MODULE} module and schema-salad-tool +## install : install the cwltool module with the "deps" dependency (galaxy-tool-util) install: FORCE pip install .[deps] -## dev : install the ${MODULE} module in dev mode +## dev : install the cwltool module in dev mode dev: install-dep pip install -e .[deps] -## dist : create a module package for distribution +## dist : create a module package for distribution dist: dist/${MODULE}-$(VERSION).tar.gz check-python3: @@ -76,11 +79,11 @@ check-python3: dist/${MODULE}-$(VERSION).tar.gz: check-python3 $(SOURCES) python setup.py sdist bdist_wheel -## docs : make the docs +## docs : make the docs docs: FORCE cd docs && $(MAKE) html -## clean : clean up all temporary / machine-generated files +## clean : clean up all temporary / machine-generated files clean: check-python3 FORCE rm -f ${MODULE}/*.pyc tests/*.pyc *.so ${MODULE}/*.so rm -Rf ${MODULE}/__pycache__/ @@ -89,7 +92,7 @@ clean: check-python3 FORCE rm -f diff-cover.html # Linting and code style related targets -## sorting imports using isort: https://github.com/timothycrosley/isort +## sort_import : sorting imports using isort: https://github.com/timothycrosley/isort sort_imports: $(PYSOURCES) mypy-stubs isort $^ @@ -97,29 +100,29 @@ remove_unused_imports: $(PYSOURCES) autoflake --in-place --remove-all-unused-imports $^ pep257: pydocstyle -## pydocstyle : check Python docstring style +## pydocstyle : check Python docstring style pydocstyle: $(PYSOURCES) pydocstyle --add-ignore=D100,D101,D102,D103 $^ || true pydocstyle_report.txt: $(PYSOURCES) pydocstyle setup.py $^ > $@ 2>&1 || true -## diff_pydocstyle_report : check Python docstring style for changed files only +## diff_pydocstyle_report : check Python docstring style for changed files only diff_pydocstyle_report: pydocstyle_report.txt diff-quality --compare-branch=main --violations=pydocstyle --fail-under=100 $^ -## codespell : check for common mispellings +## codespell : check for common mispellings codespell: codespell -w $(shell git ls-files | grep -v cwltool/schemas | grep -v cwltool/jshint/ | grep -v mypy-stubs) -## format : check/fix all code indentation and formatting (runs black) +## format : check/fix all code indentation and formatting (runs black) format: black --exclude cwltool/schemas setup.py cwltool.py cwltool tests mypy-stubs format-check: black --diff --check --exclude cwltool/schemas setup.py cwltool.py cwltool tests mypy-stubs -## pylint : run static code analysis on Python code +## pylint : run static code analysis on Python code pylint: $(PYSOURCES) pylint --msg-template="{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}" \ $^ -j0|| true @@ -154,18 +157,18 @@ diff-cover: coverage.xml diff-cover.html: coverage.xml diff-cover --compare-branch=main $^ --html-report $@ -## test : run the ${MODULE} test suite +## test : run the ${MODULE} test suite test: check-python3 $(PYSOURCES) python -m pytest -rs ${PYTEST_EXTRA} -## testcov : run the ${MODULE} test suite and collect coverage +## testcov : run the ${MODULE} test suite and collect coverage testcov: check-python3 $(PYSOURCES) python -m pytest -rs --cov --cov-config=.coveragerc --cov-report= ${PYTEST_EXTRA} sloccount.sc: $(PYSOURCES) Makefile sloccount --duplicates --wide --details $^ > $@ -## sloccount : count lines of code +## sloccount : count lines of code sloccount: $(PYSOURCES) Makefile sloccount $^