Migration to Jupyterlab (#4830)

Description of changes:
- migrate to JupyterLab (fixes #4789)
- bump Sphinx version requirements
- use `python3 -m pip` in place of the deprecated `pip3` command
- use `requirements.txt` as a constraint file to avoid version number inconsistencies with the user guide

Author: kodiakhq[bot]

.github/workflows/push_pull.yml

@@ -35,7 +35,7 @@ jobs:
   debian:
     runs-on: ubuntu-latest
     container:
-      image: ghcr.io/espressomd/docker/debian:fbdf2f2f9d76b761c0aa1308f17fb17e38501850-base-layer
+      image: ghcr.io/espressomd/docker/debian:339903979196fd7e72127f2cb5bfb27759d129f9-base-layer
       credentials:
          username: ${{ github.actor }}
          password: ${{ secrets.github_token }}
@@ -74,7 +74,7 @@ jobs:
     runs-on: ubuntu-latest
     if: ${{ github.repository == 'espressomd/espresso' }}
     container:
-      image: ghcr.io/espressomd/docker/ubuntu-wo-dependencies:fbdf2f2f9d76b761c0aa1308f17fb17e38501850-base-layer
+      image: ghcr.io/espressomd/docker/ubuntu-wo-dependencies:339903979196fd7e72127f2cb5bfb27759d129f9-base-layer
       credentials:
          username: ${{ github.actor }}
          password: ${{ secrets.github_token }}

.gitlab-ci.yml

@@ -1,4 +1,4 @@
-image: ghcr.io/espressomd/docker/ubuntu-22.04:fbdf2f2f9d76b761c0aa1308f17fb17e38501850
+image: ghcr.io/espressomd/docker/ubuntu-22.04:339903979196fd7e72127f2cb5bfb27759d129f9
 
 stages:
   - prepare
@@ -144,7 +144,7 @@ no_rotation:
 fedora:36:
   <<: *global_job_definition
   stage: build
-  image: ghcr.io/espressomd/docker/fedora:fbdf2f2f9d76b761c0aa1308f17fb17e38501850
+  image: ghcr.io/espressomd/docker/fedora:339903979196fd7e72127f2cb5bfb27759d129f9
   variables:
      with_cuda: 'false'
      with_gsl: 'false'
@@ -390,6 +390,7 @@ installation:
     - bash maintainer/CI/build_cmake.sh
     - cd build
     - make install
+    - cmake . -D ESPRESSO_BUILD_TESTS=ON
     # get path of installed files
     - CI_INSTALL_DIR="/tmp/espresso-unit-tests"
     - CI_INSTALL_PYTHON_PATH=$(dirname $(find "${CI_INSTALL_DIR}/lib" -name espressomd))
@@ -398,9 +399,9 @@ installation:
     - cp -r "src/python/object_in_fluid" "${CI_INSTALL_PYTHON_PATH}/object_in_fluid"
     # run all tests with the installed files
     - sed -i "s|$(pwd)/pypresso|${CI_INSTALL_DIR}/bin/pypresso|" testsuite/{python,scripts/samples,scripts/tutorials}/CTestTestfile.cmake
-    - make -j ${CI_CORES} check_python_skip_long
-    - make -j ${CI_CORES} check_samples
-    - make -j 2 check_tutorials
+    - make check_python_skip_long
+    - make check_samples
+    - make check_tutorials
   tags:
     - espresso
     - cuda

doc/sphinx/conf.py.in

@@ -24,7 +24,7 @@ sys.path.insert(0, '@CMAKE_BINARY_DIR@/src/python')
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
-needs_sphinx = '2.3'
+needs_sphinx = '3.5'
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom

doc/sphinx/installation.rst

@@ -175,7 +175,7 @@ To generate the Sphinx documentation, install the following packages:
 
 .. code-block:: bash
 
-    pip3 install --user -c requirements.txt \
+    python3 -m pip install --user -c requirements.txt \
         sphinx sphinxcontrib-bibtex sphinx-toggleprompt
 
 To generate the Doxygen documentation, install the following packages:
@@ -197,40 +197,36 @@ To run the samples and tutorials, start by installing the following packages:
 
 The tutorials are written in the
 `Notebook Format <https://nbformat.readthedocs.io/en/latest/>`__
-:cite:`kluyver16a` version <= 4.4 and can be executed by any of these tools:
+:cite:`kluyver16a` version 4.5 and can be executed by any of these tools:
 
-* `Jupyter Notebook <https://jupyter-notebook.readthedocs.io/en/stable/notebook.html>`__
 * `JupyterLab <https://jupyterlab.readthedocs.io/en/stable/>`__
-* `IPython <https://ipython.org/>`__ (not recommended)
 * `VS Code Jupyter <https://github.com/microsoft/vscode-jupyter>`__
+* `Jupyter Notebook <https://jupyter-notebook.readthedocs.io/en/stable/notebook.html>`__
+* `IPython <https://ipython.org/>`__ (not recommended)
 
 To check whether one of them is installed, run these commands:
 
 .. code-block:: bash
 
-    jupyter notebook --version
     jupyter lab --version
+    jupyter notebook --version
     ipython --version
     code --version
 
 If you don't have any of these tools installed and aren't sure which one
-to use, we recommend installing the historic Jupyter Notebook, since the
-|es| tutorials have been designed with the ``exercise2`` plugin in mind.
-
-To use Jupyter Notebook, install the following packages:
+to use, we recommend installing JupyterLab:
 
 .. code-block:: bash
 
-    pip3 install --user 'nbformat==5.1.3' 'nbconvert==6.4.5' 'notebook==6.4.8' 'jupyter_contrib_nbextensions==0.5.1'
-    jupyter contrib nbextension install --user
-    jupyter nbextension enable rubberband/main
-    jupyter nbextension enable exercise2/main
+    python3 -m pip install --user -c requirements.txt \
+        nbformat nbconvert jupyterlab
 
-Alternatively, to use JupyterLab, install the following packages:
+If you prefer the look and feel of Jupyter Classic, install the following:
 
 .. code-block:: bash
 
-    pip3 install --user nbformat notebook jupyterlab
+    python3 -m pip install --user -c requirements.txt \
+        nbformat nbconvert jupyterlab nbclassic
 
 Alternatively, to use VS Code Jupyter, install the following extensions:
 
@@ -316,7 +312,7 @@ Run the following commands:
       doxygen gsl numpy scipy ipython jupyter freeglut
     brew install hdf5-mpi
     brew link --force cython
-    pip install -c requirements.txt PyOpenGL matplotlib
+    python -m pip install -c requirements.txt PyOpenGL matplotlib
 
 .. _Quick installation:
 

doc/sphinx/running.rst

@@ -6,8 +6,19 @@ Running a simulation
 |es| is implemented as a Python module. This means that you need to write a
 python script for any task you want to perform with |es|. In this chapter,
 the basic structure of the interface will be explained. For a practical
-introduction, see the tutorials, which are also part of the
-distribution.
+introduction, see the tutorials, which are also part of the distribution.
+
+Most users should consider building and then installing |es| locally.
+In this way, |es| behaves like any regular Python package and will
+be recognized by the Python interpreter and Jupyter notebooks.
+
+Most developers prefer the ``pypresso`` resp. ``ipypresso`` wrapper scripts,
+which export the build folder into the ``$PYTHONPATH`` environment variable
+and then call ``python`` resp. ``jupyter``. They also introduce extra command
+line options to help developers run simulations inside a debugger.
+Command line examples in this chapter use the wrapper scripts instead of the
+Python and Jupyter programs, although they are perfectly interchangeable
+when not using a debugger.
 
 .. _Running es:
 
@@ -71,40 +82,25 @@ in the build folder, do:
 
     make tutorials
 
-The tutorials contain solutions hidden with the ``exercise2`` NB extension.
-Since this extension is only available for Jupyter Notebook, JupyterLab
-users need to convert the tutorials:
-
-.. code-block:: bash
-
-    for f in doc/tutorials/*/*.ipynb; do
-      ./pypresso doc/tutorials/convert.py exercise2 --to-jupyterlab ${f}
-    done
-
-Likewise, VS Code Jupyter users need to convert the tutorials:
-
-.. code-block:: bash
-
-    for f in doc/tutorials/*/*.ipynb; do
-      ./pypresso doc/tutorials/convert.py exercise2 --to-vscode-jupyter ${f}
-    done
+The tutorials contain solutions hidden inside disclosure boxes.
+Click on "Show solution" to reveal them.
 
 To interact with notebooks, move to the directory containing the tutorials
 and call the ``ipypresso`` script to start a local Jupyter session.
 
-For Jupyter Notebook and IPython users:
+For JupyterLab users:
 
 .. code-block:: bash
 
     cd doc/tutorials
-    ../../ipypresso notebook
+    ../../ipypresso lab
 
-For JupyterLab users:
+For Jupyter Classic users:
 
 .. code-block:: bash
 
     cd doc/tutorials
-    ../../ipypresso lab
+    ../../ipypresso nbclassic
 
 For VS Code Jupyter users, no action is needed if ``pypresso`` was set as
 the interpreter path (see details in :ref:`Running inside an IDE`).
@@ -129,29 +125,15 @@ will exit the Python interpreter and Jupyter will notify you that the current
 Python kernel stopped. If a cell takes too long to execute, you may interrupt
 it with the stop button.
 
-Solutions cells are created using the ``exercise2`` plugin from nbextensions.
-To prevent solution code cells from running when clicking on "Run All", these
-code cells need to be converted to Markdown cells and fenced with `````python``
-and ```````.
+Solutions cells are marked up with the code comment ``# SOLUTION CELL``
+(must be on the first line). In the build folder, these solution cells
+will be automatically converted to Markdown cells.
 
 To close the Jupyter session, go to the terminal where it was started and use
 the keyboard shortcut Ctrl+C twice.
 
-When starting a Jupyter session, you may see the following warning in the
-terminal:
-
-.. code-block:: none
-
-    [TerminalIPythonApp] WARNING | Subcommand `ipython notebook` is deprecated and will be removed in future versions.
-    [TerminalIPythonApp] WARNING | You likely want to use `jupyter notebook` in the future
-
-This only means |es| was compiled with IPython instead of Jupyter. If Jupyter
-is installed on your system, the notebook will automatically close IPython and
-start Jupyter. To recompile |es| with Jupyter, provide ``cmake`` with the flag
-``-D IPYTHON_EXECUTABLE=$(which jupyter)``.
-
-You can find the official Jupyter documentation at
-https://jupyter.readthedocs.io/en/latest/running.html
+You can find the official JupyterLab documentation at
+https://jupyterlab.readthedocs.io/en/latest/user/interface.html
 
 .. _Running inside an IDE:
 

doc/tutorials/CMakeLists.txt

@@ -62,31 +62,23 @@ function(NB_EXPORT)
         "${NB_FILE};${NB_EXPORT_ADD_SCRIPTS};${CMAKE_BINARY_DIR}/doc/tutorials/convert.py;${CMAKE_BINARY_DIR}/testsuite/scripts/importlib_wrapper.py"
       COMMAND
         ${CMAKE_BINARY_DIR}/pypresso
-        ${CMAKE_BINARY_DIR}/doc/tutorials/convert.py ci --execute --exercise2
-        --remove-empty-cells --input ${NB_FILE} --output ${NB_FILE_RUN}
+        ${CMAKE_BINARY_DIR}/doc/tutorials/convert.py ci --execute
+        --prepare-for-html --input ${NB_FILE} --output ${NB_FILE_RUN}
         --substitutions ${NB_EXPORT_VAR_SUBST} --scripts
         ${NB_EXPORT_ADD_SCRIPTS})
   else()
     set(NB_FILE_RUN ${NB_FILE})
   endif()
 
   add_custom_command(
-    OUTPUT ${HTML_FILE}
-    DEPENDS ${NB_FILE_RUN};${NB_EXPORT_ADD_SCRIPTS}
-    COMMAND
-      ${CMAKE_BINARY_DIR}/pypresso ${CMAKE_BINARY_DIR}/doc/tutorials/convert.py
-      ci --exercise2 --input ${NB_FILE_RUN} --output ${NB_FILE_RUN}~
+    OUTPUT ${HTML_FILE} DEPENDS ${NB_FILE_RUN};${NB_EXPORT_ADD_SCRIPTS}
     COMMAND ${IPYTHON_EXECUTABLE} nbconvert --to "html" --output ${HTML_FILE}
-            ${NB_FILE_RUN}~)
+            ${NB_FILE_RUN})
 
   add_custom_command(
-    OUTPUT ${PY_FILE}
-    DEPENDS ${NB_FILE}
-    COMMAND
-      ${CMAKE_BINARY_DIR}/pypresso ${CMAKE_BINARY_DIR}/doc/tutorials/convert.py
-      ci --exercise2 --input ${NB_FILE} --output ${NB_FILE}~
+    OUTPUT ${PY_FILE} DEPENDS ${NB_FILE}
     COMMAND ${IPYTHON_EXECUTABLE} nbconvert --to "python" --output ${PY_FILE}
-            ${NB_FILE}~)
+            ${NB_FILE})
 
   add_custom_target("${NB_EXPORT_TARGET}_html" DEPENDS ${HTML_FILE}
                                                        ${DEPENDENCY_OF_TARGET})

doc/tutorials/Readme.md

@@ -62,7 +62,7 @@ physical systems.
 * **Electrodes**  
   Modelling electrodes and measuring differential capacitance with the ELC method.  
   [Part 1](electrodes/electrodes_part1.ipynb) |
-  Part 2 (work in progress)
+  [Part 2](electrodes/electrodes_part2.ipynb)
 * **Constant pH method**  
   Modelling an acid dissociation curve using the constant pH method.  
   [Guide](constant_pH/constant_pH.ipynb)
@@ -91,32 +91,12 @@ in the build folder, do:
 make tutorials
 ```
 
-The tutorials contain solutions hidden with the ``exercise2`` extension.
-Since this extension is only available for Jupyter Notebook, JupyterLab
-users need to convert the tutorials:
-
-```sh
-for f in doc/tutorials/*/*.ipynb; do
-  ./pypresso doc/tutorials/convert.py exercise2 --to-jupyterlab ${f};
-done
-```
-
 All tutorials can be viewed with their solutions
 [online](https://espressomd.github.io/doc/tutorials.html).
 
 ### Running the tutorials interactively
 
-To view the tutorials, first change to the tutorials directory and then run
-the `ipypresso` script from the directory into which espresso was compiled.
-
-For Jupyter Notebook and IPython users:
-
-```sh
-cd doc/tutorials
-../../ipypresso notebook
-```
-
-For JupyterLab users:
+To view the tutorials in the build folder, run the following commands:
 
 ```sh
 cd doc/tutorials