CONTRIBUTING.md, README.md: Update documentation for contributors

.actrc

@@ -0,0 +1,13 @@
+--workflows .github/act-serial.yaml
+--platform js-20.04=ghcr.io/catthehacker/ubuntu:js-20.04
+--platform ubuntu-20.04=ghcr.io/catthehacker/ubuntu:act-20.04
+--platform ubuntu-22.04=ghcr.io/catthehacker/ubuntu:act-22.04
+--platform ubuntu-latest=ghcr.io/catthehacker/ubuntu:act-latest
+
+# If using podman, use one of these, preferably in your ~/.actrc:
+# For act-cli.rpm on Fedora 37:
+# --container-daemon-socket /run/user/1000/podman/podman.sock
+# For the latest act, use this in your ~/.actrc:
+# --container-daemon-socket unix:///run/user/1000/podman/podman.sock
+# More information on setting up act can be found at:
+# https://github.com/xenserver/python-libs/blob/master/CONTRIBUTING.md#running-github-actions-locally-using-act

CONTRIBUTING.md

@@ -0,0 +1,196 @@
+## Development setup on Fedora 37
+
+On Fedora 37, the `tox` rpm installs all Python versions.
+But this `tox` is older, so install `tox==4.5.1` using `pip` (see below)
+
+```bash
+sudo dnf install tox;sudo rpm -e tox
+```
+
+## Development setup on Ubuntu 22.04
+
+```bash
+sudo apt update
+sudo apt install software-properties-common python{2,3}-dev
+sudo add-apt-repository -y ppa:deadsnakes/ppa
+sudo apt-get install -y python3.{8,11}{,-distutils}
+```
+
+Installation of additional python versions for testing different versions:
+
+- If `deadsnakes/ppa` does not work, e.g. for Python 3.6, `conda` or `pyenv` can be used.
+  For instructions, see https://realpython.com/intro-to-pyenv:
+  ```yaml
+  sudo apt install -y build-essential xz-utils zlib1g-dev
+                      lib{bz2,ffi,lzma,readline,ssl,sqlite3}-dev
+  curl https://pyenv.run | bash  # add displayed commands to .bashrc
+  pyenv install 3.{6,8,11} && pyenv local 3.{6,8,11} # builds them
+  ```
+- For testing on newer Ubuntu which has `python2-dev`, but not `pip2`, install `pip2` this way:
+  ```yml
+  curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py;sudo python2 get-pip.py
+  ```
+
+You may want to install `pytype` in your user environment to run it directly without `tox`:
+
+```bash
+# On Python != 3.8, pytype can't import xml.dom.minidom, use 3.8:
+python3.8 -m pip install pytype
+python -m pip install tabulate
+./run-pytype.py
+```
+
+## Installation of dependencies using `pip`
+
+### Testing locally and in GitHub CI using `tox`
+
+This project uses `tox` to run the tests for different python versions. Intro:
+
+> _"Managing a Project's Virtual environments with `tox` -
+> A comprehensive beginner's introduction to `tox`":_
+> https://www.seanh.cc/2018/09/01/tox-tutorial/
+
+`tox` runs `pytest`, `pylint` and static analysis using `mypy`, `pyre`, `pytype`, and `pyright`.
+Links:
+
+- https://mypy.readthedocs.io/en/stable/
+- https://microsoft.github.io/pyright/
+- https://google.github.io/pytype/
+- https://pyre-check.org/
+
+With `tox`, developers can run the full test suite for Python 2.7 and 3.x.
+The same test suite is used in GitHub CI:
+
+```bash
+pip3 install --user --upgrade 'py>=1.11.0' 'virtualenv<20.22' 'tox==4.5.1'; hash -r; tox
+```
+
+Explanation:
+
+- `tox>=4` is needed in order to fix reading the python2.7 dependencies from `pyproject.toml`
+- `tox==4.5.1` is the last version not depending on `virtualenv>=20.23` (breaks Python 2.7)
+- The latest versions of `tox` need `'py>=1.11.0'`. This ensures that it is at least `1.11`.
+- `virtualenv-20.22` breaks using python2.7 for the `py27`, so has to be `virtualenv<20.22`.
+
+## Installation of all development dependencies
+
+Using pip-tools, you can extract the requirements and extras from `pyptoject.toml`:
+
+```bash
+PYTHON=python3.10
+EXTRAS=.,test,mypy,pyre,pytype,tox
+PFLAGS="--no-warn-conflicts --resolver=backtracking"
+$PYTHON -m pip install pip-tools
+$PYTHON -m piptools compile --extra=$EXTRAS -o - pyproject.toml |
+    $PYTHON -m pip install -r /dev/stdin $PFLAGS
+```
+
+With this, you can run most of the CI tests run by `tox` and GitHub CI also from the shell.
+
+You can use e.g.: `tox -e py27-test -e py3.10-covcombine-check`
+The syntax is `-e py<python-version>-<factor1>[-factor2]`.
+A few of the factors are:
+
+- `test`: runs `pytest`
+- `cov`: runs `pytest --cov` and generates `XML` and `HTML` reports in `.tox/py<ver>-cov/logs/`
+- `check`: runs `mypy`
+- `fox`: runs like `cov` but then opens the `HTML` reports in Firefox
+
+## Recommended `tox` and `pytest` plugins for development
+
+When updating existing tests or developing new code with new test coverage, we might want to
+ignore all other tests. This can be achieved with an exciting plugin called `pytest-picked`:
+`pytest --picked` will collect all test modules that were newly created or changed but not
+yet committed in a Git repository and run only them.
+
+`pytest-sugar` is a plugin that, once installed, automatically changes the format of the
+`pytest` standard output to include a graphical %-progress bar when running the test suite.
+
+For nicer diffs of dictionaries, arrays and the like, use `pytest-clarity` or `pytest-icdiff`:
+
+```py
+pip install "pytest<7" pytest-picked pytest-sugar pytest-clarity # pytest-icdiff
+```
+
+To verify or extract the dependencies and extras configured in `pyproject.toml` and `tox.ini`
+for specific `tox` environments, you can use https://pypi.org/project/tox-current-env/:
+
+```bash
+tox --print-deps-to=pytype-deps.txt --print-extras-to=pytype-extras.txt -e pytype
+```
+
+For more information to debug `pytest` test suites see: https://stribny.name/blog/pytest/
+
+## Running GitHub actions locally using `act`
+
+- `act` uses `docker` (also mimicked by `podman-docker`) to run GitHub actions locally
+- While `act` does not use the same GitHub runner images, they are similar.
+
+### `act` using `podman` with `podman-docker` instead of `docker`
+
+This allows to use a user-local `podman` daemon in a `systemd ---user` session.
+Thus, the containers and images are local to the development user.
+With it, multiple users can use it on the same host without interfering with each other.
+The `podman` daemon and all `pods` are completely unprivileged and rootless.
+
+Fedora 37:
+
+```bash
+sudo dnf install podman-docker
+# Note: If possible, don't install the act-cli.rpm because it is old and it
+# needs different configuration, e.g. the unix:// prefix needs to be omitted!
+```
+
+Ubuntu 22.04:
+
+```bash
+sudo apt-get install -y podman-docker
+```
+
+Install `act` as `~/.local/bin/act`
+
+```bash
+curl -L https://github.com/nektos/act/releases/latest/download/act_Linux_x86_64.tar.gz|
+    tar xfz - -C ~/.local/bin
+```
+
+To run `podman` as your user, run these as your user:
+
+```bash
+systemctl enable --now --user podman.socket # Only configures the podman socket
+systemctl start --user podman.socket        # Start the docker-compatible unix socket
+# Ubuntu only, Fedora 37 configures it already with more unqualifies search registries:
+echo 'unqualified-search-registries = ["docker.io"]' | sudo tee -a /etc/containers/registries.conf
+sudo touch /etc/containers/nodocker         # Quiet the docker emulation notification
+echo "--container-daemon-socket unix://$XDG_RUNTIME_DIR/podman/podman.sock" >>~/.actrc
+```
+
+In any case, you should test the `docker` interface now:
+
+```bash
+docker run -it --rm alpine:latest grep NAME /etc/os-release
+```
+
+## Recommendations for Windows and WSL2
+
+### Using Docker on WSL2/22.04
+
+Ubuntu 22.04 LTS uses `iptables-nft` by default.
+Switch to `iptables-legacy` so that Docker will work:
+https://crapts.org/2022/05/15/install-docker-in-wsl2-with-ubuntu-22-04-lts/
+
+### Copy selection on selecting test (without need for Ctrl-C)
+
+On traditional X11 and KDE Plasma, selected text is automatically copied
+to the X selection/clipboard for pasting it. To use this engrained behavior
+on Windows as well, it seems the only reliable way to have it for all apps
+is a `AutoHotKey` script:
+
+- https://www.ilovefreesoftware.com/30/tutorial/auto-copy-selected-text-clipboard-windows-10.html
+
+While individual extensions for VS Code, Firefox, chrome do work partially,
+they either don't cover the Firefox URL bar, the VS Code terminal and so on:
+
+- https://addons.mozilla.org/en-GB/firefox/addon/copy-on-select
+- https://marketplace.visualstudio.com/items?itemName=dinhani.copy-on-select (VS Code)
+- https://www.jackofalladmins.com/knowledge%20bombs/dev%20dungeon/windows-terminal-copy-selection/

README.md

@@ -56,76 +56,25 @@ open a recent workflow run the latest and scroll down until you see the tables!
 - `.github/act-serial.yaml`: Configuration for the jobs run by the local GitHub actions runner `act`
 - `pylintrc`: Configuration file of `Pylint`
 
-## Testing locally and in GitHub CI using `tox`
+## Installation and Setup of a development environment
 
-`pytest` runs tests, checks by `pylint` and `mypy`. With `tox`, developers can
-run the full test suite for Python 2.7 and 3.x. Unit tests are passing, but there are
-many Python3 issues which it does not uncover yet.
+For the installation of the general development dependencies, visit [INSTALL.md](INSTALL.md)
 
-> _"Intro: Managing a Project's Virtualenvs with tox -
-> A comprehensive beginner's introduction to tox":_
-> https://www.seanh.cc/2018/09/01/tox-tutorial/
+### Updating tests using `pytest-watch` (`ptw`)
 
-To run the tests for all supported and installed python versions, run:
-
-```yaml
-pip3 install --user --upgrade 'py>=1.11.0' 'virtualenv<20.22' 'tox>=4.5.1'; hash -r; tox
-```
-
-- `tox>=4` is needed in order to fix reading the python2.7 deps from `pyproject.toml`
-- The latest versions of `tox` need `'py>=1.11.0'`. Ensure that it is at least 1.11.
-- `virtualenv-20.22` breaks using python2.7 for the `py27` virtualenv with tox,
-  therefore it has to be downgraded thus `'virtualenv<20.22'`.
-
-Using pip-tools, you can also extract the requirements and extras from `pyptoject.toml`:
-
-```bash
-PYTHON=python3.10
-$PYTHON -m pip install pip-tools
-$PYTHON -m piptools compile --extra=.,test,mypy,pyre,pytype,tox -o - pyproject.toml |
-    $PYTHON -m pip install -r /dev/stdin --no-warn-conflicts
-```
-
-With this, you can run most of the CI tests run by `tox` and GitHub CI also from the shell.
-
-You can run `tox` with just the Python versions you have using `tox -e py27-test -e py3.11-mypy`.
-The syntax is `-e py<pvthon-version>-<factor1>[-factor2]` The currently supported factors
-are:
-
-- `test`: runs `pytest`
-- `cov`: runs `pytest --cov` and generates `XML` and `HTML` reports in `.tox/py<ver>-cov/logs/`
-- `mypy`: runs `mypy`
-- `fox`: runs like `cov` but then opens the `HTML` reports in Firefox!
-
-## Recommended `pytest` plugins for development
-
-When changing existing tests or developing new code with new test coverage, we might want to
-ignore all other tests. This can be achieved with an exciting plugin called `pytest-picked`:
-`pytest --picked` will collect all test modules that were newly created or changed but not
-yet committed in a Git repository and run only them.
-
-`pytest-sugar` is a plugin that, once installed, automatically changes the format of the
-`pytest` standard output to include a graphical %-progress bar when running the test suite.
-
-For nicer diffs of dictionaries, arrays and the like, use `pytest-clarity` or `pytest-icdiff`.
-For more information to debug pytest test suites see: https://stribny.name/blog/pytest/
-
-```py
-pip install "pytest<7" pytest-picked pytest-sugar pytest-clarity
-```
+- `pip install pytest-watch` - `ptw` watches changed files and runs `pytest` after changes are saved.
+  - Then run `ptw` on the code/tests you work on, e.g.: `ptw tests/test_pci_*` and edit the files.
 
 ## Example development workflow
 
-- `pip install pytest-watch` - `ptw` watches changed files and runs pytest after changes are saved.
-  - Then run `ptw` on the code/tests you work on, e.g.: `ptw tests/test_pci_*` and edit the files.
 - Run the tests for at also with `LC_ALL=C python3.6 -m pytest` to check for any `ascii` codec
   issues by Python3.6
 - Test with `python2.7 -m pytest`
 - Run `mypy` (without any arguments - The configuration is in `pyproject.toml`)
 - Run `./run-pytype.py`
 - Run `./run-pyre.py`
 - Run `tox -e py36-lint` and fix any `Pylint` warnings
-- Run `tox -e py38-covcombine` and fix any missing diff-coverage.
+- Run `tox -e py310-covcombine-check` and fix any missing diff-coverage.
 - Run `tox` for the full CI test suite
 - Run `act` for the full CI test suite in local containers (similar to GitHub action containers)
 - Commit with `--signoff` on a new branch and push it and check the triggered GitHub Action run succeeds.
@@ -136,38 +85,21 @@ The list of `virtualenvs` configured in tox can be shown using this command: `to
 ```yaml
 $ tox -av
 default environments:
-py311-pyre      -> Run in a py311 virtualenv: Run pyre for static analyis, only passes using: tox -e py311-pyre
-py310-pytype    -> Run in a py310 virtualenv: Run pytype for static analyis, intro: https://youtu.be/abvW0mOrDiY
-py39-check      -> Run in a py39 virtualenv: Run mypy for static analyis
-py38-covcombine -> Run in a py38 virtualenv: Generate combined coverage reports with py27-test coverage merged
-py37-mdreport   -> Run in a py37 virtualenv: Make a test report (which is shown in the GitHub Actions Summary Page)
-py36-lint       -> Run in a py36 virtualenv: Run pylint and fail on warnings remaining on lines in the diff to master
+py36-lint              -> Run in a py36 virtualenv: Run pylint and fail on warnings remaining on lines in the diff to master
+py311-pyre             -> Run in a py311 virtualenv: Run pyre for static analyis, only passes using: tox -e py311-pyre
+py38-pytype            -> Run in a py38 virtualenv: Run pytype for static analyis, intro: https://youtu.be/abvW0mOrDiY
+py310-covcombine-check -> Run in a py310 virtualenv: Generate combined coverage reports with py27-test coverage merged Run mypy for static analyis
 
 additional environments:
-cov             -> Run in a /usr/bin/python3 virtualenv: Generate coverage html reports (incl. diff-cover) for this environment
-covcp           -> Run in a /usr/bin/python3 virtualenv: Copy the generated .converage and coverage.xml to the UPLOAD_DIR dir
-fox             -> Run in a /usr/bin/python3 virtualenv: Generate combined coverage html reports and open them in firefox
-test            -> Run in a /usr/bin/python3 virtualenv: Run pytest in this environment with --cov for use in other stages
+cov                    -> Run in a python virtualenv: Generate coverage html reports (incl. diff-cover) for this environment
+covcp                  -> Run in a python virtualenv: Copy the generated .converage and coverage.xml to the UPLOAD_DIR dir
+fox                    -> Run in a python virtualenv: Generate combined coverage html reports and open them in firefox
+mdreport               -> Run in a python virtualenv: Make a test report (which is shown in the GitHub Actions Summary Page)
+test                   -> Run in a python virtualenv: Run pytest in this environment with --cov for use in other stages
 ```
 
 If you have only one version of Python3, that works too. Use: `tox -e py<ver>-test`
 
-Installation of additional python versions for testing different versions:
-
-- Fedora 37: `sudo dnf install tox` installs all Python versions, even 3.12a7.
-- On Ubuntu, the deadsnakes/ppa is broken(except for 3.12), so conda or pyenv has to be used.
-  For full instructions, see https://realpython.com/intro-to-pyenv/, E.g install on Ubuntu:
-  ```yaml
-  sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev
-                          libreadline-dev libsqlite3-dev xz-utils libffi-dev liblzma-dev
-  curl https://pyenv.run | bash # and add the displayed commands to .bashrc
-  pyenv install 3.{6,7,8,9} && pyenv local 3.{6,7,8,9} # builds and adds them
-  ```
-- For testing on newer Ubuntu hosts which have `python2-dev`, but not `pip2`, install `pip2` this way:
-  ```yml
-  curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py;sudo python2 get-pip.py
-  ```
-
 ## Static analysis using mypy, pyre, pyright and pytype
 
 The preconditions for using static analysis with `mypy` (which passes now but has