Black vs. Click fix + test, (dev)docs, etc. (#325)

* DEV/DEP: update note about black + q2 vsns

* BLD: test on two additional Q2 versions

And document supported versions better in the CI.

* REL: update changelog

* DEV/BLD: Document and catch Q2 CLI breaking thing

ha -- so it doesn't impact the tests but it does impact the CLI (i
shoulda documented this clearer when i first saw it), and now this
will be reflected in the build. This build should now fail.

* DEV: fix up dev docs re: q2 vsn

* DOC: update redsea screenshot

* DEP: fix black --> click --> qiime2 conflict

* STY: Re-format python using black 21.12b0

Since we pinned it in this PR. Shoulda done this with the previous
commit, whoops

* DEV: note about dev environment

* DEP: Omit Black from dev deps; update dev docs+CI

I think this is one of the more painless solutions to this problem.
Eesh.

* REL: note black vs. click issue in chglog

* DEV: update makefile

(forgor)

* DOC: update README & CHANGELOG re: requirements

* REL: changelog tense, etc.

Author: fedarko

.github/workflows/qiime2.yml

@@ -12,7 +12,19 @@ jobs:
       matrix:
         # Test on multiple QIIME 2 versions. This is a nice way of making sure
         # that Qurro works on all of the QIIME 2 versions it actually, you
-        # know, supports.
+        # know, supports. As of writing, this includes QIIME 2 versions
+        #
+        # 1. 2020.11
+        # 2. 2021.2
+        # 3. 2021.4
+        # 4. 2021.8
+        # 5. 2021.11
+        # 6. 2022.2
+        # 7. 2022.8
+        #
+        # We currently test just a subset of these versions, because each of
+        # these builds takes a while (and testing each of these would probs be
+        # overkill).
         #
         # We could use a matrix of just QIIME 2 version strings (e.g.
         # "2020.11", ...), but other stuff about the URLs change across QIIME 2
@@ -25,6 +37,8 @@ jobs:
         # for the heads up about this :)
         q2-yaml: [
           "https://raw.githubusercontent.com/qiime2/environment-files/master/2020.11/release/qiime2-2020.11-py36-linux-conda.yml",
+          "https://raw.githubusercontent.com/qiime2/environment-files/master/2021.8/release/qiime2-2021.8-py38-linux-conda.yml",
+          "https://raw.githubusercontent.com/qiime2/environment-files/master/2022.2/release/qiime2-2022.2-py38-linux-conda.yml",
           "https://raw.githubusercontent.com/qiime2/environment-files/master/latest/staging/qiime2-latest-py38-linux-conda.yml"
         ]
 
@@ -57,8 +71,16 @@ jobs:
       - name: Run tests
         run: conda run -n qiime2-dev make test
 
-      - name: Lint and stylecheck
-        run: conda run -n qiime2-dev make stylecheck
+      - name: Lint and stylecheck (not using Black to check Python code)
+        run: conda run -n qiime2-dev make stylecheck_not_black
+
+      # It's possible to install packages that break QIIME 2's CLI (e.g.
+      # installing Click >= 8 into a Q2 2022.2 environment seems to break it --
+      # this is discussed in the developer documentation, CONTRIBUTING.md).
+      # Verify that, by installing Qurro, we have not "broken" QIIME 2 in this
+      # way.
+      - name: Test that QIIME 2's CLI works
+        run: conda run -n qiime2-dev qiime info
 
       - name: Upload code coverage information to Codecov
         uses: codecov/codecov-action@v2

.github/workflows/standalone.yml

@@ -26,20 +26,24 @@ jobs:
 
       - name: Install Qurro (and its pip dependencies)
         run: pip install -e .[dev]
+
+      - name: Install Black, also (no QIIME 2 to worry about conflicts with)
+        run: pip install "black >= 22.3.0"
 
       - name: Set up Node.js
         uses: actions/setup-node@v1
 
       - name: Install Qurro's Node.js development dependencies
         run: npm install -g mocha-headless-chrome nyc prettier@2.0.5 jshint
 
-      # Just run the tests; don't bother running stylechecking / linting, since
-      # that's already done in the QIIME 2 CI.
       - name: Run tests
         run: |
           make pytest_standalone
           make jstest
 
+      - name: Lint and stylecheck (including using Black to check Python code)
+        run: make stylecheck
+
       # NOTE: For now, we only submit coverage reports from the QIIME 2 CI,
       # since the tests that that CI runs are a superset of those ran by this
       # "standalone" CI. (Also, Codecov wasn't merging them, which caused

CHANGELOG.md

@@ -1,9 +1,9 @@
 # Qurro changelog
 
-## Qurro 0.8.0 (date TBD)
+## Qurro 0.8.0 (October 19, 2022)
 ### Features added
-- Add an option to Qurro's visualization interface to exclude metadata columns
-  from the "sample plot data" output TSV
+- Added an option to Qurro's visualization interface to exclude metadata
+  columns from the "sample plot data" output TSV
   ([#306](https://github.com/biocore/qurro/issues/306),
   [#313](https://github.com/biocore/qurro/pull/313)).
   - This should make it easier to merge the sample log-ratios exported from
@@ -12,34 +12,45 @@
     [Gemelli](https://github.com/biocore/gemelli)'s tutorials.)
 ### Backward-incompatible changes
 - Switched the required pandas version from `>= 0.24.0, <1` to `>= 1`.
-  This makes Qurro now compatible with QIIME 2 versions `>= 2020.11`!
+  **This makes Qurro now compatible with QIIME 2 versions `>= 2020.11`!**
   - However, this change removes support for older QIIME 2 versions. (It may be
     possible to get this version of Qurro installed into an older QIIME 2
-    environment, but we do not explicitly support this.)
+    environment, but we do not explicitly support this.) If you need to use
+    Qurro in older QIIME 2 environments, you can install an older version of
+    Qurro (e.g. `pip install "qurro==0.7.1"`).
   - Notably, Songbird has not yet made the shift to pandas `>= 1`, so Songbird
     and Qurro will need to be installed into separate environments; the "Red
     Sea" example notebook details this process.
+- Updated a few other dependency versions to fix various problems (e.g.
+  explicitly requiring SciPy and pinning it to `scipy < 1.9.0` to fix
+  [this scikit-bio issue](https://github.com/biocore/scikit-bio/issues/1818),
+  removing Black from the development dependencies for reasons discussed in
+  [`CONTRIBUTING.md`](https://github.com/biocore/qurro/blob/master/CONTRIBUTING.md)).
 ### Bug fixes
 ### Performance enhancements
 ### Miscellaneous
-- Improve various parts of Qurro's code to remove various warnings (for
+- Updated various parts of Qurro's code to remove various warnings (for
   example, about certain things being deprecated).
-- Update Qurro's example notebooks:
+- Updated Qurro's example notebooks:
   - Updated the "Red Sea" notebook to explain how to use
     Songbird and Qurro in different conda environments.
   - Updated the "ALDEx2" notebook to explain how we recommend installing ALDEx2
     nowadays.
 - Updated the documentation to refer to the published version of the Mackerel
   data's paper ([Minich et al. 2020](https://journals.asm.org/doi/full/10.1128/mSphere.00401-20)).
+- Fixed various broken links in the documentation
+  ([#318](https://github.com/biocore/qurro/pull/318),
+  [#320](https://github.com/biocore/qurro/pull/320)).
 - Ported Qurro's continuous integration from Travis CI to GitHub Actions
   ([#316](https://github.com/biocore/qurro/issues/316)).
 - Improved Qurro's continuous integration in multiple ways:
   - Test on multiple QIIME 2 versions.
   - Test the standalone Qurro functionality on multiple Python versions.
   - Test the standalone Qurro functionality in a non-QIIME-2 environment
     (similar to EMPress' "standalone" CI).
-- Updated the development documentation regarding the minimum QIIME 2 version,
-  incompatibilities between `black` and `click`, etc.
+- Updated the development documentation
+  ([`CONTRIBUTING.md`](https://github.com/biocore/qurro/blob/master/CONTRIBUTING.md))
+  regarding the minimum QIIME 2 version, dependency version issues, etc.
 
 ## Qurro 0.7.1 (May 22, 2020)
 ### Features added

CONTRIBUTING.md

@@ -10,16 +10,25 @@ in this repository or email the Qurro development team
 
 ## Setting up a development environment
 
+We'll use a QIIME 2 environment as the basis for creating our "development
+environment" -- this is convenient because some of Qurro's tests rely on QIIME
+2 being installed. (You could also create a development environment that does
+not use QIIME 2, some of Qurro's tests would not be able to be run in such an
+environment.)
+
 1. Fork Qurro.
 1. Clone your fork of Qurro's source code to your computer.
 2. Create a development conda environment for Qurro:
     1. Install the latest version of QIIME 2 natively,
-       [as you would normally](https://docs.qiime2.org/2022.2/install/native/).
-       _You'll need to install a QIIME 2 version of at least 2020.11._
+       [as you would normally](https://docs.qiime2.org/2022.8/install/native/).
+       _You'll need to install a QIIME 2 version of at least 2022.8._
     2. In a terminal, navigate to the folder to which you cloned your fork of
        Qurro's source code above. Run `pip install -e .[dev]` inside this folder to
        install Qurro along with its normal and development Python dependencies.
-    3. Install the various Node.js requirements for testing Qurro's JavaScript
+    3. Install Black: `pip install "black >= 22.3.0"`. We don't include this in
+       the development dependencies for reasons that are explained later in
+       this document.
+    4. Install the various Node.js requirements for testing Qurro's JavaScript
        code. This can be done by running
        `npm install -g mocha-headless-chrome jshint prettier@2.0.5 nyc`. Note that
        this will install these programs globally on your system.
@@ -53,22 +62,24 @@ generated by Qurro's Python tests, and is used to populate some of the
 JavaScript tests. Long story short, you'll just need to run `make test` (or
 just `make pytest`) before running `make jstest`.
 
-### When running QIIME 2, Qurro, etc. I get `TypeError: __init__() got multiple values for argument 'obj'`
+### When running `qiime`, `qiime qurro`, etc. from the CLI I get `TypeError: __init__() got multiple values for argument 'obj'`
 
-This happens due to a [Click](https://click.palletsprojects.com/en/8.1.x/)
+This happens due to a [Click](https://click.palletsprojects.com/)
 version being installed that is unsupported by
 QIIME 2. See [this q2cli issue](https://github.com/qiime2/q2cli/issues/259) for
 context. This can come up in development because one of Qurro's development
-dependencies is
-[Black](https://github.com/psf/black), and later versions of Black can rely
-on Click 8 (and as of writing QIIME 2 doesn't seem to support Click 8 yet).
-
-If you run into this error, you can probably get around it by installing an
-older version of Black, although earlier versions of Click might disagree with
-the later versions that we use in the CI. There isn't a great solution for this
-yet that I know of (short of pinning Black to an earlier version and
-reformatting all the Python code accordingly...), but hopefully when QIIME 2
-supports Click 8 the problem will vanish.
+dependencies (used for auto-formatting and format-checking the Python code) is
+[Black](https://github.com/psf/black), and later versions of Black rely
+on Click 8 (and versions of QIIME 2 before 2022.8 do not support Click 8).
+
+Pinning to an older version of Black causes another problem, because -- in this
+case -- the newer version of Click used in QIIME 2 2022.8 environments will
+break Black (with [this error](https://github.com/psf/black/issues/2964)).
+
+So, we get around this issue by removing Black from our development
+dependencies completely. If you installed a version of QIIME 2 of at least
+2022.8, then installing Black (in the directions above) should not cause
+problems (or at least it should not cause these particular problems).
 
 ## Acknowledgements
 

Makefile

@@ -32,11 +32,15 @@ jstest:
 	nyc instrument qurro/support_files/js/ qurro/tests/web_tests/instrumented_js/
 	mocha-headless-chrome -f qurro/tests/web_tests/index.html -c js_coverage.json
 
+stylecheck: stylecheck_not_black stylecheck_black
+
+stylecheck_black:
+	black --check -l 79 qurro/ setup.py
+
 # Assumes this is being run from the root directory of the qurro repo
 # (since that's where the .jshintrc is located).
-stylecheck:
+stylecheck_not_black:
 	flake8 --ignore=E203,W503 qurro/ setup.py
-	black --check -l 79 qurro/ setup.py
 	jshint $(JSLOCS)
 	prettier --check --tab-width 4 $(JSLOCS) $(HTMLCSSLOCS)
 

README.md

@@ -58,11 +58,7 @@ Qurro can be used standalone (as a Python 3 script that generates a
 folder containing a HTML/JS/CSS visualization) or as a
 [QIIME 2](https://qiime2.org/) plugin (that generates a QZV file that can be
 visualized at [view.qiime2.org](https://view.qiime2.org/) or by using
-`qiime tools view`). *Starting with Qurro v0.8.0, Qurro requires a QIIME 2
-version of at least 2020.11.*
-
-Qurro should work with most modern web browsers. Firefox or Chrome are
-recommended.
+`qiime tools view`).
 
 Qurro is still being developed, so backwards-incompatible changes might
 occur. If you have any bug reports, feature requests, questions, or if you just
@@ -85,7 +81,18 @@ can be viewed online [here](https://biocore.github.io/qurro/demos/red_sea/index.
 
 ## Installation and Usage
 
-You can install Qurro using [pip](https://pip.pypa.io/en/stable/) or [conda](https://docs.conda.io/en/latest/). In either case, a python version of at least 3.6 and less than 3.10 is required to use Qurro.
+You can install Qurro using [pip](https://pip.pypa.io/en/stable/)
+or [conda](https://docs.conda.io/en/latest/).
+
+### System requirements
+**If you're using Qurro within QIIME 2,** you will need a QIIME 2 version of at
+least 2020.11.
+
+**If you're using Qurro outside of QIIME 2,** you will need a Python version of
+at least 3.6 and less than 3.10.
+
+In either case, Qurro should work with most modern web browsers; Firefox or Chrome are
+recommended.
 
 ### Installing with `pip`
 ```bash
@@ -203,8 +210,11 @@ For python testing/style checking, Qurro uses
 [pytest-cov](https://github.com/pytest-dev/pytest-cov),
 [flake8](http://flake8.pycqa.org/en/latest/), and
 [black](https://github.com/ambv/black). You'll also need to have QIIME 2
-installed to run most of the python tests (your QIIME 2 version should be at
-least 2020.11; this is the version which switched to pandas >= 1).
+installed to run most of the python tests (note that, due to click vs. black
+vs. QIIME 2 dependency issues, you should use a QIIME 2 environment of at least
+2022.8; see
+[`CONTRIBUTING.md`](https://github.com/biocore/qurro/blob/master/CONTRIBUTING.md)
+for details).
 
 For JavaScript testing/style checking, Qurro uses
 [Mocha](https://mochajs.org/), [Chai](https://www.chaijs.com/),

screenshots/redsea_data.png

@@ -100,9 +100,6 @@
             "pytest >= 4.2",
             "pytest-cov >= 2.0",
             "flake8",
-            # If we don't pin black to being at least 22.3.0, then it and click
-            # will conflict w/ an error: https://github.com/psf/black/pull/2966
-            "black >= 22.3.0",
             "nbconvert",
         ]
     },