-
Notifications
You must be signed in to change notification settings - Fork 129
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a topic on jupyter-lmod in documentation #640
base: main
Are you sure you want to change the base?
Changes from all commits
8bc87ff
fcdc839
e933d07
7a21881
c1381e0
50efcf3
3b11c1c
339cc86
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
@@ -0,0 +1,130 @@ | ||||||||||||||||||
Jupyter and Lmod | ||||||||||||||||||
================ | ||||||||||||||||||
|
||||||||||||||||||
It is possible to use Lmod inside of Jupyter with the terminal or jupyter-lmod. | ||||||||||||||||||
|
||||||||||||||||||
jupyter-lmod is a frontend and server extension for Jupyter (notebook and jupyterlab). | ||||||||||||||||||
The extension graphically provides most of Lmod's functions: | ||||||||||||||||||
- Browser and filter available modules | ||||||||||||||||||
- Load and unload modules | ||||||||||||||||||
- Display a module content | ||||||||||||||||||
- Create and load collections | ||||||||||||||||||
|
||||||||||||||||||
The extension is located in the 'Softwares' tab of the main interface in Jupyter Notebook, | ||||||||||||||||||
while in JupyterLab, it can be accessed as a sidepanel by clicking on the Lmod logo icon | ||||||||||||||||||
located in the sidebar. | ||||||||||||||||||
|
||||||||||||||||||
Installation | ||||||||||||||||||
~~~~~~~~~~~~ | ||||||||||||||||||
|
||||||||||||||||||
To install jupyter-lmod extension: :: | ||||||||||||||||||
|
||||||||||||||||||
pip install jupyterlmod | ||||||||||||||||||
cmd-ntrf marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||||||||||
|
||||||||||||||||||
This enables the extension for both Jupyter Notebook and JupyterLab. | ||||||||||||||||||
|
||||||||||||||||||
How does it work? | ||||||||||||||||||
~~~~~~~~~~~~~~~~~ | ||||||||||||||||||
|
||||||||||||||||||
jupyter-lmod uses lmod python interface to generate Python code | ||||||||||||||||||
that modifies the environment variables of the Python process running Jupyter. All child | ||||||||||||||||||
processes of Jupyter (including kernels) created after Lmod calls inherit | ||||||||||||||||||
the environment variables as defined by the set of loaded modules. | ||||||||||||||||||
cmd-ntrf marked this conversation as resolved.
Show resolved
Hide resolved
Comment on lines
+29
to
+32
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This sentence is a bit confusing, I suggest to simplify it:
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
In concrete terms, kernels in use by active notebooks will not be aware of changes | ||||||||||||||||||
made to the environment (e.g. **$PATH**, **$PYTHONPATH**, **$LD_LIBRARY_PATH**) by modules | ||||||||||||||||||
loaded after they were spawned. Depending on the Jupyter version, a manual kernel restart | ||||||||||||||||||
or a complete kernel shutdown/start cycle is needed to make an active notebook aware of changes | ||||||||||||||||||
to the environment variables. | ||||||||||||||||||
|
||||||||||||||||||
Lmod environment variable configuration for jupyter-lmod | ||||||||||||||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||||||||||||||||||
|
||||||||||||||||||
The plugin requires the environment variable **$LMOD_CMD** to be defined when Jupyter starts, | ||||||||||||||||||
plus all environment variables required by Lmod to work properly (i.e.: **$MODULEPATH**). | ||||||||||||||||||
|
||||||||||||||||||
Some JupyterHub spawners do not setup the complete user's environment, leaving **$LMOD_CMD** | ||||||||||||||||||
and **$MODULEPATH** undefined. To address this issue, the environment variables have to be | ||||||||||||||||||
defined in **jupyterhub_config.py** using | ||||||||||||||||||
`c.Spawner.environment <https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.environment>`_. | ||||||||||||||||||
|
||||||||||||||||||
If the environment variables are defined before JupyterHub launch, you can add their | ||||||||||||||||||
names only to `c.Spawner.env_keep <https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.env_keep>`_ | ||||||||||||||||||
instead. | ||||||||||||||||||
Comment on lines
+51
to
+53
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Technically these variable are only needed in the single-user server. JupyterHub itself does not need them.
Suggested change
|
||||||||||||||||||
|
||||||||||||||||||
Controlling launcher items with jupyter-lmod | ||||||||||||||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||||||||||||||||||
|
||||||||||||||||||
jupyter-lmod can add a kernel or a server proxy to the Jupyter launcher | ||||||||||||||||||
following module loading if the module matches some specifications. | ||||||||||||||||||
|
||||||||||||||||||
kernels | ||||||||||||||||||
+++++++ | ||||||||||||||||||
|
||||||||||||||||||
If a module adds one or more absolute paths to **$JUPYTER_PATH**, Jupyter's kernel spec manager | ||||||||||||||||||
will be notified of the changes. If the added paths contain kernel spec files (e.g: **kernel.json**), | ||||||||||||||||||
the kernels will be added to the available choices in Jupyter interface. | ||||||||||||||||||
|
||||||||||||||||||
The following example demonstrates this feature with Julia. Here, we have an **ijulia-kernel/1.8** module | ||||||||||||||||||
that prepends a path to **$JUPYTER_PATH**. :: | ||||||||||||||||||
|
||||||||||||||||||
whatis("Description: IJulia kernelspec for Julia 1.8") | ||||||||||||||||||
whatis("Homepage: https://julialang.org/") | ||||||||||||||||||
whatis("URL: https://julialang.org/") | ||||||||||||||||||
conflict("ijulia-kernel") | ||||||||||||||||||
depends_on("julia/1.8.1") | ||||||||||||||||||
depends_on("ijulia/.1.24.0") | ||||||||||||||||||
prepend_path("JUPYTER_PATH","/software/Core/ijulia-kernel/1.8/share/jupyter") | ||||||||||||||||||
|
||||||||||||||||||
|
||||||||||||||||||
In **/software/Core/ijulia-kernel/1.8/share/jupyter/kernels/julia-1.8**, there are four files :: | ||||||||||||||||||
|
||||||||||||||||||
kernel.json logo-32x32.png logo-64x64.png logo-svg.svg | ||||||||||||||||||
|
||||||||||||||||||
and **kernel.json** looks like this :: | ||||||||||||||||||
|
||||||||||||||||||
{ | ||||||||||||||||||
"display_name": "Julia 1.8.1", | ||||||||||||||||||
"argv": [ | ||||||||||||||||||
"/software/Core/julia/1.8.1/bin/julia", | ||||||||||||||||||
"-i", | ||||||||||||||||||
"--color=yes", | ||||||||||||||||||
"/software/Core/ijulia/1.24.0/packages/IJulia/6TIq1/src/kernel.jl", | ||||||||||||||||||
"{connection_file}" | ||||||||||||||||||
], | ||||||||||||||||||
"language": "julia", | ||||||||||||||||||
"env": {}, | ||||||||||||||||||
"interrupt_mode": "signal" | ||||||||||||||||||
} | ||||||||||||||||||
|
||||||||||||||||||
The folders and files under **share/jupyter/kernels** are typically generated by a kernelspec installer. | ||||||||||||||||||
|
||||||||||||||||||
|
||||||||||||||||||
server proxies | ||||||||||||||||||
++++++++++++++ | ||||||||||||||||||
|
||||||||||||||||||
Jupyter Server Proxy is an extension for Jupyter and JupyterLab that let users run arbitrary | ||||||||||||||||||
external processes (such as RStudio, Shiny Server, Syncthing, PostgreSQL, Code Server, etc) | ||||||||||||||||||
alongside their notebook server and provide authenticated web access to them using a path | ||||||||||||||||||
like /rstudio next to others like /lab. It also provides buttons in the launcher to launch | ||||||||||||||||||
these external processes. | ||||||||||||||||||
|
||||||||||||||||||
When jupyter-lmod and jupyter-server-proxy are both installed, it is possible for the launcher | ||||||||||||||||||
button of an external process to appear only when its corresponding module is loaded. | ||||||||||||||||||
In order to do so, the frontend extension of jupyter-server-proxy must be deactivated as | ||||||||||||||||||
the control of the launcher will be entrusted to jupyter-lmod. :: | ||||||||||||||||||
|
||||||||||||||||||
jupyter nbextension disable --py jupyter_server_proxy --sys-prefix | ||||||||||||||||||
jupyter labextension disable @jupyterlab/server-proxy | ||||||||||||||||||
jupyter labextension disable jupyterlab-server-proxy | ||||||||||||||||||
|
||||||||||||||||||
Then, the name of each `server proxy <https://jupyter-server-proxy.readthedocs.io/en/latest/server-process.html#server-process-options>` | ||||||||||||||||||
must match the name of module. When a module with a name corresponding to a server | ||||||||||||||||||
proxy name is loaded, jupyter-lmod will add a button to the launcher to start that process. | ||||||||||||||||||
|
||||||||||||||||||
References | ||||||||||||||||||
~~~~~~~~~~ | ||||||||||||||||||
|
||||||||||||||||||
For more information, refer to jupyter-lmod github repo: | ||||||||||||||||||
https://www.github.com/cmd-ntrf/jupyter-lmod | ||||||||||||||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe add a link to your repo from the start 🙂