Fixes for newer Jenkins version. Require Python 3.9+.

Workaround for https://issues.jenkins.io/browse/JENKINS-63845 Some timing issue fiexs in test. Major test update to move from tox to nox. Get rid of Tenjin to generate coverage settings, use environment variables in coverage_rc instead. TODO: More cleanup to get rid of tmp_intall file copying.
lhupfeldt · Oct 10, 2024 · fce07dc · fce07dc
1 parent 6ff1063
commit fce07dc
Show file tree

Hide file tree

Showing 52 changed files with 782 additions and 590 deletions.
diff --git a/.gitignore b/.gitignore
@@ -25,6 +25,7 @@ pip-log.txt
 .coverage*
 coverage_rc.tenjin.cache*
 .tox
+.nox
 nosetests.xml
 .pytest_cache/
 

diff --git a/.travis.yml b/.travis.yml
@@ -3,39 +3,24 @@ python:
   - 3.6
   - 3.7
   - 3.8
-  - pypy3.6-7.0.0
-matrix:
-    include:
-        - python: 3.7
-          dist: xenial
-          env: REQUIREMENTS=lowest
-        - python: 3.7
-          dist: xenial
-          env: REQUIREMENTS=release
-        - python: 3.8
-          dist: xenial
-          env: REQUIREMENTS=lowest
-        - python: 3.8
-          dist: xenial
-          env: REQUIREMENTS=release
-    allow_failures:
-        - python: pypy3.6-7.0.0
-script: tox
+  - 3.9
 os:
   - linux
 before_install:
   - travis_retry pip install --upgrade pip
   - travis_retry pip install requirements-builder
   - requirements-builder --level=min setup.py > .travis-lowest-requirements.txt
   - requirements-builder --level=pypi setup.py > .travis-release-requirements.txt
-install:
-  - pip install tox-travis
-  - pip install coveralls
-  - travis_retry pip install -r .travis-$REQUIREMENTS-requirements.txt
-  - pip install -e .
 env:
   - REQUIREMENTS=lowest
   - REQUIREMENTS=release
+install:
+  - pip install --upgrade nox
+  - pip install --upgrade coveralls
+  - travis_retry pip install -r .travis-$REQUIREMENTS-requirements.txt
+  - pip install -e .
+script:
+  - nox
 after_success:
   # rcfile name duplicated in test/run.py
   - coveralls --rcfile="./test/.coverage_rc_mock_script"
diff --git a/INSTALL.md b/INSTALL.md
@@ -3,147 +3,146 @@ Installation
 
 Python 3.6 or later is required.
 A recent Jenkins is required.
-Hudson may be supported, but it is not longer tested.
 
 1. The easy way:
-   Install python-devel (required by the psutil dependency of the script_api)
+   Install *python-devel* (required by the *psutil* dependency of the *script_api*)
    E.g on fedora:
-   sudo dnf install python-devel
 
-   pip install --user -U .
+        sudo dnf install python-devel
+        pip install --user --upgrade jenkinsflow
 
    To uninstall:
-   pip uninstall jenkinsflow
 
-   Read the file demo/demo_security.py if you have security enabled your Jenkins.
+        pip uninstall jenkinsflow
 
-Jenkinsflow uses it's own specialized 'jenkins_api' python module to access Jenkins, using the Jenkins rest api.
+   Read the file *demo/demo_security.py* if you have security enabled your Jenkins.
+
+Jenkinsflow uses it's own specialized *jenkins_api* python module to access Jenkins, using the Jenkins rest api.
 
 2. Manually:
 2.1. Install dependencies:
-   pip install requests click atomicfile
 
-   optional: pip install tenjin (if you want to use the template based job loader)
+        pip install requests click atomicfile
+
+   optional: `pip install tenjin` (if you want to use the template based job loader)
 
 2.2. Install dependencies for experimental features:
-   To use the experimental script api:
-     Install python-devel (see above)
-     pip install psutil setproctitle
+   To use the experimental script API install python-devel (see above)
+
+       pip install psutil setproctitle
 
    To use the experimental visualisation feature:
-     pip install bottle
 
-2.3. Read the file demo/demo_security.py for notes about security, if you have enabled security on your Jenkins
+       pip install bottle
+
+2.3. Read the file *demo/demo_security.py* for notes about security, if you have enabled security on your Jenkins
 
 2.4. All set! You can now create jobs that have a shell execution step, which will a use this library to control the running of other jobs.
    See the demo directory for example flows. The demo jobs can be loaded by running tests, see below.
 
 
 Note: I think jenkinsflow should work on Windows, but it has not been tested.
-   I'm SURE the test/run.py script will fail on Windows. There are a few Linux/Unix bits in the test setup. Check test/framework/config.py and
-   test/tmp_install.sh. Patches are welcome:)
+   I'm SURE the tests will fail on Windows. There are a few Linux/Unix bits in the test setup. Check *test/framework/cfg/* and
+   *test/framework/tmp_install.sh*. Patches are welcome:)
 
 
 Test
 ----
 
-0. The mocked and script_api test can be run using 'tox'
-   pip install tox
+0. The mocked and script_api test can be run using `nox`
 
-1. The tests which actually run Jenkins jobs currently do not run under tox.
+        pip install nox
 
-   Install pytest and tenjin template engine:
-   pip install --user -U -r test/requirements.txt
-
-   # The test will also test the generation of documentation, for this you need:
-   pip install --user -U -r doc/requirements.txt
-
-
-2. Important Jenkins setup and test preparation:
+1. Important Jenkins setup and test preparation:
    Configure security -
       Some of the tests requires security to be enabled.
       You need to create two users in Jenkins:
-      Read the file demo/demo_security.py and create the user specified.
-      Create a user called 'jenkinsflow_authtest1', password 'abcæøåÆØÅ'. ( u'\u00e6\u00f8\u00e5\u00c6\u00d8\u00c5' )
+      Read the file *demo/demo_security.py* and create the user specified.
+      Create a user called **jenkinsflow_authtest1**, password **abcæøåÆØÅ**. ( u'\u00e6\u00f8\u00e5\u00c6\u00d8\u00c5' )
 
    Set the number of executers -
-      Jenkins is default configured with only two executors on master. To avoid timeouts in the test cases this must be raised to at least 32.
+      Jenkins is default configured with only two executors on 'built-in' node. To avoid time-outs in the test cases this must be raised to at least **32**.
       This is necessary because some of the test cases will execute a large amount of jobs in parallel.
 
    Change the 'Quite period' -
-      Jenkins is default configured with a 'Quiet period' of 5 seconds. To avoid timeouts in the test cases this must be set to 0.
+      Jenkins is default configured with a 'Quiet period' of 5 seconds. To avoid time-outs in the test cases this must be set to **0**.
 
    Set Jenkins URL -
      Jenkins: Manage Jenkins -> Configure System -> Jenkins Location -> Jenkins URL
 
-     The url should not use 'localhost'
+     The URL should **not** use **localhost**.
 
    Your Jenkins needs to be on the host where you are running the test. If it is not, you will need to make jenkinsflow available to Jenkins. See
-   test/tmp_install.sh
+   *test/framework/tmp_install.sh*
 
 3. Run the tests:
-   Use tox
-   or
-   Use ./test/run.py to run the all tests.
 
-   JENKINS_URL=<your Jenkins> python ./test/run.py --mock-speedup=100 --direct-url <non proxied url different from JENKINS_URL>
+   Use `nox`
+
+        JENKINS_URL=<your Jenkins> nox --mock-speedup=100 --direct-url <non proxied URL different from JENKINS_URL>
 
    Note: you may omit JENKINS_URL if your Jenkins is on http://localhost:8080.
-   Note: you may omit --direct-url if your Jenkins is on http://localhost:8080
+   Note: you may omit --direct-url if your Jenkins is on http://localhost:8080.
 
-   The test script will run the test suite with mocked jenkins api, script_api and jenkins_api in parallel. The mocked api is a very fast test of the flow logic.
-   Mocked tests and script_api tests do not require Jenkins.
+   Note: Run `nox -- --help` and look at *custom options:* to see special options.
+
+   The test script will run the test suite with *mocked jenkins_api*, *script_api* and *jenkins_api* in parallel. The mocked api is a very fast test of the flow logic.
+   Mocked tests and script-api tests do not require Jenkins.
    The test jobs are automatically created in Jenkins.
 
-   It is possible to select a subset of the apis using the --apis option.
+   It is possible to select a subset of the apis using the `--api` option.
 
-   The value given to --mock-speedup is the time speedup for the mocked tests. If you have a reasonably fast computer, try 2000.
-   If you get FlowTimeoutException try a lower value.
-   If you get "<job> is expected to be running, but state is IDLE" try a lower value.
+   The value given to `--mock-speedup` is the time speedup for the mocked tests. If you have a reasonably fast computer, try **2000**.
+   If you get `FlowTimeoutException` try a lower value.
+   If you get *\<job\> is expected to be running, but state is IDLE* try a lower value.
 
    By default tests are run in parallel using xdist and jobs are not deleted (but will be updated) before each run.
-   You should have 32 executors or more for this, the cpu/disk load will be small, as the test jobs don't really do anything except sleep.
-   To disable the use of xdist use --job-delete.
-
-   All jobs created by the test script are prefixed with 'jenkinsflow_', so they can easily be removed.
+   You should have **32** executors or more for this, the CPU/disk load will be small, as the test jobs don't really do anything except sleep.
+   To disable the use of xdist use `--job-delete`.
 
-   The test suite creates jobs called ..._0flow. These jobs are not executed by the test suite, by you can run them to see what the flows look like in a Jenkins job.
-   If your Jenkins is not secured, you must set username and password to '' in demo_security,  in order to be able to run all the ..._0flow jobs.
+   All jobs created by the test script are prefixed with **jenkinsflow_**, so they can be easily removed.
 
-   To see more test options, run ./test/run.py ---help
+   The test suite creates jobs called *..._0flow*. These jobs are not executed by the test suite, by you can run them to see what the flows look like in a Jenkins job.
+   If your Jenkins is not secured, you must set username and password to '' in *demo_security.py*,  in order to be able to run all the ..._0flow jobs.
 
 
 Demos
-----
+-----
 
 1. Run tests as described above to load jobs into Jenkins
 
 2. Demo scripts can be executed from command line:
    python ./demo/<demo>.py
 
 3. Demo scripts can be executed from the loaded 'jenkinsflow_demo__<demo-name>__0flow' Jenkins jobs.
-   Jenkins needs to be able to find the scripts, the run.py script creates a test installation.
+   Jenkins needs to be able to find the scripts, executing `nox` creates a test installation.
 
 
 Flow Graph Visualisation
------------------------
+------------------------
+
+1. To see a flow graph of the basic demo in your browser, execute:
 
-1. To see a flow graph of the basic demo in your browser:
-   Start 'python ./visual/server.py' --json-dir '/tmp/jenkinsflow-test/graphs/jenkinsflow_demo__basic' before running ./demo/basic.py
-   Open localhost:9090 in your browser.
+        python ./visual/server.py' --json-dir '/tmp/jenkinsflow-test/graphs/jenkinsflow_demo__basic
 
-   The test suite also puts some other graps in subdirectories under '/tmp/jenkinsflow-test/graphs'.
-   The 'visual' feature is still experimental and does not yet show live info about the running flow/jobs.
+   before running *./demo/basic.py*
 
-   If you run ...0flow jobs that generate graphs from Jenkins the json graph file will be put in the workspace.
+   Open http://localhost:9090 in your browser.
+
+   The test suite also puts some other graphs in subdirectories under */tmp/jenkinsflow-test/graphs*.
+   The *visual* feature is still experimental and does not yet show live info about the running flow/jobs.
+
+   If you run *...0flow* jobs that generate graphs from Jenkins the json graph file will be put in the workspace.
 
 
 Documentation
 ----
 
 1. Install sphinx and extensions:
-   pip install 'sphinx>=1.6' sphinxcontrib-programoutput
+
+        pip install 'sphinx>=1.6' sphinxcontrib-programoutput
 
 2. Build documentation:
-   cd doc/source
-   make html (or some other format supported by sphinx)
+
+        cd doc/source
+        make html (or some other format supported by sphinx)
diff --git a/README.rst b/README.rst
@@ -4,13 +4,12 @@ jenkinsflow
 ===========
 
 Python API with high level build flow constructs (parallel/serial) for
-Jenkins (and Hudson). Allows full scriptable control over the execution
+Jenkins. Allows full scriptable control over the execution
 of Jenkins jobs. Also allows running 'jobs' without using Jenkins (for
 testing without reloading Jenkins jobs).
 
 See INSTALL.md for installation and test setup. See demo/... for some
-usage examples. I don't test continuously on Hudson, but patches are
-welcome.
+usage examples.
 
 Thanks to Aleksey Maksimov for contributing various bits, including the
 graph visualization.

diff --git a/cli/cli.py b/cli/cli.py
@@ -15,15 +15,16 @@
     __package__ = "jenkinsflow.cli"
 
 
-from .set_build_description import set_build_description
+from .set_build_description import set_build_description, set_build_description_hidden
 
 
 @click.group()
 def cli():
     """Commandline utilities for jenkinsflow"""
 
 
-cli.add_command(set_build_description, name="set_build_description")
+cli.add_command(set_build_description)
+cli.add_command(set_build_description_hidden)  # Backwards compatibility
 
 
 if __name__ == "__main__":

diff --git a/cli/set_build_description.py b/cli/set_build_description.py
@@ -6,25 +6,30 @@
 from jenkinsflow.utils import set_build_description as usbd
 
 
-@click.command()
-@click.option('--description', help="The description to set on the build")
-@click.option('--replace/--no-replace', default=False, help="Replace existing description, if any, instead of appending.")
-@click.option('--separator', default='\n', help="A separator to insert between any existing description and the new 'description' if 'replace' is not specified.")
-@click.option('--username', help="User Name for Jenkin authentication with secured Jenkins")
-@click.option('--password', help="Password of Jenkins User")
-@click.option('--build-url', help='Build URL', envvar='BUILD_URL')
-@click.option('--job-name', help='Job Name', envvar='JOB_NAME')
-@click.option('--build-number', help="Build Number", type=click.INT, envvar='BUILD_NUMBER')
-@click.option(
-    '--direct-url',
-    default=None,
-    help="Jenkins URL - preferably non-proxied. If not specified, the value of JENKINS_URL or HUDSON_URL environment variables will be used.")
-def set_build_description(description, replace, separator, username, password, build_url, job_name, build_number, direct_url):
-    """Utility to set/append build description on a job build.
-
-    When called from a Jenkins job you can leave out the '--build-url', '--job-name' and '--build-number' arguments, the BUILD_URL env variable will be used.
-    """
-
-    # %(file)s --job-name <job_name> --build-number <build_number> --description <description> [--direct-url <direct_url>] [--replace | --separator <separator>] [(--username <user_name> --password <password>)]
-
-    usbd.set_build_description(description, replace, separator, username, password, build_url, job_name, build_number, direct_url)
+# Generate twice for backwards compatibility, first (hidden) is the old name
+for name, hidden in (("set_build_description", True), ("set-build-description", False)):
+    @click.command(name=name, hidden=hidden)
+    @click.option('--description', help="The description to set on the build")
+    @click.option('--replace/--no-replace', default=False, help="Replace existing description, if any, instead of appending.")
+    @click.option('--separator', default='\n', help="A separator to insert between any existing description and the new 'description' if 'replace' is not specified.")
+    @click.option('--username', help="User Name for Jenkin authentication with secured Jenkins")
+    @click.option('--password', help="Password of Jenkins User")
+    @click.option('--build-url', help='Build URL', envvar='BUILD_URL')
+    @click.option('--job-name', help='Job Name', envvar='JOB_NAME')
+    @click.option('--build-number', help="Build Number", type=click.INT, envvar='BUILD_NUMBER')
+    @click.option(
+        '--direct-url',
+        default=None,
+        help="Jenkins URL - preferably non-proxied. If not specified, the value of JENKINS_URL or HUDSON_URL environment variables will be used.")
+    def set_build_description(description, replace, separator, username, password, build_url, job_name, build_number, direct_url):
+        """Utility to set/append build description on a job build.
+
+        When called from a Jenkins job you can leave out the '--build-url', '--job-name' and '--build-number' arguments, the BUILD_URL env variable will be used.
+        """
+
+        # %(file)s --job-name <job_name> --build-number <build_number> --description <description> [--direct-url <direct_url>] [--replace | --separator <separator>] [(--username <user_name> --password <password>)]
+
+        usbd.set_build_description(description, replace, separator, username, password, build_url, job_name, build_number, direct_url)
+
+    if hidden:
+        set_build_description_hidden = set_build_description
diff --git a/demo/basic.py b/demo/basic.py
@@ -7,7 +7,7 @@
 
 from jenkinsflow.flow import serial
 
-import get_jenkins_api
+from jenkinsflow.demo import get_jenkins_api
 
 
 def main(api, securitytoken):

diff --git a/demo/calculated_flow.py b/demo/calculated_flow.py
@@ -12,7 +12,7 @@
 from jenkinsflow.flow import serial
 from jenkinsflow.unbuffered import UnBuffered
 
-import get_jenkins_api
+from jenkinsflow.demo import get_jenkins_api
 
 
 # Unbuffered output does not work well in Jenkins/Hudson, so in case

diff --git a/demo/errors.py b/demo/errors.py
@@ -5,7 +5,7 @@
 
 from jenkinsflow.flow import serial
 
-import get_jenkins_api
+from jenkinsflow.demo import get_jenkins_api
 
 
 def main(api, securitytoken):

diff --git a/demo/get_jenkins_api.py b/demo/get_jenkins_api.py
@@ -2,7 +2,7 @@
 
 from jenkinsflow.jenkins_api import Jenkins
 
-import demo_security as security
+from jenkinsflow.demo import demo_security as security
 
 
 def get_jenkins_api():

diff --git a/demo/hide_password.py b/demo/hide_password.py
@@ -5,7 +5,7 @@
 
 from jenkinsflow.flow import serial
 
-import get_jenkins_api
+from jenkinsflow.demo import get_jenkins_api
 
 
 def main(api, securitytoken):