build: merge from circus-tent/circus #4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Missing yield on `watcher.set_numprocesses` call was causing this coroutine to run and spawn processes while @synchronized lock was already released. So this coroutine could overlap with the next pump of `manage_watchers` and cause a lot more processes to be spawned than necessary.
The `SysHandler` class issues `quit` command when it catches the `SIGTERM` signal. This command in its turn calls `arbiter.stop()`, but if you put a breakpoint into this function you will notice that it's beign called twice, once from `quit` command and once from this weird place in controller's command dispatch callback. I digged through history and it looks like leftovers of some old command refactoring.
Making circusctl return non-zero exit code when command resulted in `staus == "error"`.
Fix unicode error with Python 2 in get_config
Watcher.reap_process() used os.waitpid(pid, os.WNOHANG) to check process status and assumed waitpid() would throw OSError with errno == errno.EAGAIN if the process was still running. This was wrong, waitpid() will return a tuple (0, 0) for living child process, so a correct way to handle child processes still running is to check for that value.
try block in Arbiter.reap_processes contains a call to os.waitpid and to Watcher.reap_process. Python's os.waitpid does not throw OSError with errno set to EAGAIN, functions called by Watcher.reap_process also should not do that - so, there is no reason to check for that errno value and handle it.
Circus can compare string reprs of config to detect a change (for example, it compares commands and stringified config is a part of a command to launch plugin). Converting to string is done by iterating over the dict, which depends on the order of items in it, which can be different for equal dicts. Sorting items of dict before putting them to string will guarantee equal config strings for equal configs.
Wait for a child to finish in Watcher.reap_process
Fix #1038 - sort string representation of plugin config
Fix #997 - reloadconfig concurrency
Fix #998 - premature shutdown leaking processes
Fixing #1001 - non-zero exit code on conflict errors
…rnado version either
This reverts commit ba13ea2.
bugfix: papa_endpoint ipc socket is ignored, tcp is created issue #1066
Lock tornado version to pre-5.0 and PyZMQ version to pre-17.0
…so restrict tornado to v<5
The original `nose` library has been abandoned and does not provide support for Python 3.9 and above. The replacement is `nose2`.
This will prevent all jobs from being stopped if one encounters an error.
Add Python 3.10 to the matrix of the CI workflow. All Python version numbers are properly wrapped in strings, otherwise they are interpreted as floats and `3.10` will become `3.1`. The `setup.py` is updated to officially record support of Python 3.10 and the check for the minimum requirement is updated to actually check for Python 3.7, it was mistakenly still checking for Python 3.5. Finally, the tests for the `papa` optional dependency are disabled because `papa` is not compatible with Python 3.10. It seems the project has been abandoned as the latest officially supported version is Python 3.4 and the last release was in early 2016.
PEP 621 introduces the `pyproject.toml` file which is now the recommended way for defining a project with its build and install dependencies. In addition, it allows to provide configuration for various tools, such as tox, instead of using individual files for each tool. The `MANIFEST.in` file is no longer required since the build backend `flit` will automatically include all files, except those explicitly marked with `[tool.flit.sdist] exclude`. The description is added to the `__init__.py` of the package and it is marked as `dynamic` in the `pyproject.toml` which will automatically fetch it from the package docstring.
The `cd.yml` is a Github Actions workflow that triggers when a new tag is pushed to the source repository. It will check that the version specified in the `__version__` attribute matches that of the tag, and if the case will run the full test suite. If all tests pass, the package will be built using `flit` and pushed to PyPI. This will significantly simplify and automate the release process.
The `papa` library provides a process kernel that `circus` allowed ot use through configuration. However, `papa` seems to be no longer maintained as its last release was in January 2016. Officially it only declares to be compatible up to Python 3.4. Apparently, so far it has still been working until Python 3.10 where it actually breaks. Since it is no longer maintained, we remove support for it from `circus`.
Also update the `actions/checkout` and `actions/setup-python` actions to v3 and v4, respectively.
The tags of `circus` do not start with `v` but are simply of the `X.Y.Z` version format.
The conditional of the `validate-release-tag` job was never matched since it contained a type on the organization using `circus-test` instead of `circus-tent`. The `ubuntu` job also always installed Python 3.9 despite using a matrix of various versions.
There were two invalid trove classifiers:
Operating System :: POSIX :: Windows
should be:
Operating System :: Microsoft :: Windows
And the license classifier had a trailing double quote. These mistakes
were raised by `flit` when trying to publish the package. Valid
classifiers are provided by: https://pypi.org/classifiers/
* `pipes` module of standard library is replaced with `shlex` * `pyzmq.utils.strtypes` was compatibility layer for Python 2 and 3. The `u` method was to convert to unicode but this is no longer applicable in Python 3 so can be removed. * Address internal deprecation warnings of `statsd` not being set to `True` in the config if a stats endpoint is defined.
This is considered good practice and allows us to easily distribute the package without shipping the tests as well.
The test scripts run by workers in the tests were importing modules from the `circus/tests` directory. This was working since the `circus` package would be automatically part of the `PYTHONPATH` when it was installed. Now that the `tests` directory is moved outside of the `circus` package however, tests were failing because imports like `tests.generic` were no longer importable. This is fixed by explicitly passing the `tests` path in the `PYTHONPATH` environment variable that is passed to the workers of the test arbiter.
The tests for MacOS run very slowly for some reason, causing the workflow to be cancelled. For now simply extending the timeout limit until the cause of the slowness is addressed.
Other versions are causing tests to fail.
These are required for the `test_venv_site_packages` in `tests/test_watcher.py`
Support tests for up to py3.13
The method to construct it from the `version_info` tuple causes the
`validate_release_tag.py` script used in the CD workflow to fail. The
abstract syntax tree parses stumbles on the `join` statement. The
following minimal example:
ast.literal_eval(ast.parse("'.'.join([])").body[0])
raises the same exception:
ValueError: malformed node or string on line 1
Since the `version_info` tuple is not actually used anywhere, the
problem is circumvented by simply defining the version string directly.
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
23 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.