-
Notifications
You must be signed in to change notification settings - Fork 652
Documentation deployment
Documentation for all MDAnalysis repositories should be self-deployed if served from the mdanalysis.org URL, instead of using ReadTheDocs (RTD). While RTD is simple and easy to use, it is not compliant with our Privacy Policy, as it adds Google Analytics and cookies (https://github.com/MDAnalysis/mdacli/issues/77). Google Analytics can be toggled off, but it's less apparent what to do about cookies.
Therefore we have rolled our own deployment, adapted from MDTraj's approach. After doing the GitHub and DNS setup and setting up conf.py, this comprises the following steps:
- Build documentation (typically done via a CI approach; formerly Travis but now Github Actions)
- Move documentation into a $version subfolder; i.e.
cd package/doc/html/html; mkdir 2.0.1; mv *.* 2.0.1
- Update
versions.json
and write or update the sitemap index - Push documentation directory to the
gh-pages
branch
Below is an explanation of the different parts of the documentation, followed by a more detailed list of steps and where they happen.
The set-up should be that the repository serves a GitHub Pages site from the gh-pages
branch. The custom domain should be set in GitHub Settings > Pages. You can use either a subdomain (e.g. docs.mdanalysis.org/) or a subdirectory (e.g. mdanalysis.org/docs). If you choose a subdirectory, no further work is required; a CNAME record for www.mdanalysis.org is already set up to point to mdanalysis.github.io. If you choose a subdomain, you need to add a new CNAME record in Cloudflare > DNS. The record should be for the subdomain part only, and point to mdanalysis.github.io
. e.g., the documentation CNAME record is from docs
-> mdanalysis.github.io
. What this means is that the contents of the gh-pages
branch is available at docs.mdanalysis.org.
A selection of core devs should have access to the Cloudflare account, which is under Oliver's ASU email address.
See https://github.com/MDAnalysis/MDAnalysis.github.io/wiki/DNS for more and please update it if more records are added.
These extensions need to be installed to build the sitemap and update the theme.
- sphinx-sitemap
- msmb_theme
- sphinx_rtd_theme
These settings need to be in conf.py
:
import msmb_theme
extensions = [..., 'sphinx_sitemap', 'sphinx_rtd_theme']
site_url = "https://docs.mdanalysis.org/"
html_theme = 'msmb_theme'
html_theme_path = [
msmb_theme.get_html_theme_path(),
sphinx_rtd_theme.get_html_theme_path()
]
html_context = {
'versions_json_url': os.path.join(site_url, "versions.json") # this should be the default I think
}
This section contains a description of different parts of the documentation, in no particular order.
The actual documentation is what gets built when you run make html
. These comprise the actual HTML files that you can open in your browser. They are usually built in the root documentation directory: package/doc/html/html
for the MDAnalysis/mdanalysis repo.
Sitemaps are XML files that are used to provide information about the pages of the documentation. These are for search engines to see what pages have to offer and do a better job of providing results. Sitemaps are built as part of the documentation, because we use the sphinx-sitemap
extension.
Sitemaps can only contain URLs within the directory. e.g. a sitemap for the 1.0.1 version cannot contain links to the 2.0.0 version. Therefore, we also need a sitemap index file. This is another XML file that contains links to the individual version sitemaps. The sitemap index file is written by us, in update_json_stubs_sitemap.py
.
Because we are after versioned documentation, we install msmb_theme for the little versions popup to navigate between the documentation. It is an add-on to sphinx_rtd_theme
, which must also be installed. The docs use versions.json
to show the individual versions. The data in versions.json
is a list of these dictionaries:
{"version": "0.20.1",
"display": "0.20.1", # this is how the version is labelled in the popup
"url": "https://docs.mdanalysis.org/0.20.1", # link to documentation for it
"latest": False # whether it's the latest version or not}
versions.json
should be in the root directory of the documentation. i.e., for docs.mdanalysis.org
, it's at https://docs.mdanalysis.org/versions.json -- you can actually click that link and view the file.
We write and edit versions.json
in update_json_stubs_sitemap.py
.
It's convenient to maintain up-to-date documentation at a constant URL for links and intersphinx. For MDAnalysis/mdanalysis, this is done at https://docs.mdanalysis.org/stable/ and https://docs.mdanalysis.org/dev/ . Just like the 1.0.1 docs are in a 1.0.1 folder, the stable and dev are just folders of documentation. These are copies of whichever version has been identified to be the latest release, and latest development version. Copying these directories is done in update_json_stubs_sitemap.py
.
An HTML redirect is a short page that immediately goes to another page. It looks like this:
<!DOCTYPE html>
<meta charset="utf-8">
<title>Redirecting to {url}</title>
<meta http-equiv="refresh" content="0; URL={url}">
<link rel="canonical" href="{url}">
We use this to redirect pages such as docs.mdanalysis.org
to docs.mdanalysis.org/stable
. This is likely all that new MDAnalysis repositories will need. However, for repositories that use to serve documentation without versions, we can also write redirect stubs for all the stable documentation.
e.g. the original MDAnalysis documentation did not have versions, so links to documentation looked like this: https://docs.mdanalysis.org/documentation_pages/topology_modules
For backwards compatibility, update_json_stubs_sitemap.py
recursively goes through all the documentation in stable
and writes redirect stubs for all of them. An example for the above page is https://github.com/MDAnalysis/mdanalysis/blob/gh-pages/documentation_pages/topology_modules.html . If you click on https://docs.mdanalysis.org/documentation_pages/topology_modules , it should redirect to https://docs.mdanalysis.org/stable/documentation_pages/topology_modules .
We also redirect from /latest/index.html
to the actual latest version, also for legacy reasons.
Most of the documentation parts are added or modified in the CI or in update_json_stubs_sitemap.py
.
These are the following steps:
- CI: build documentation (usually done as a test of whether documentation builds)
- CI: determine version number. This can be done by importing the package if it's installed, or string parsing something. Set it to $VERSION
- CI: Go into the built documentation directory. From now on this is the root directory for gh-pages. All version directories are called subdirectories.
-
CI: make a directory called $VERSION and move everything into $VERSION (
mkdir ../${VERSION} && mv * ../${VERSION} && mv ../${VERSION} $VERSION
-
CI: Fetch the
gh-pages
branch into the root directory. -
CI: Call
update_json_stubs_sitemap.py
-
Python: Download
versions.json
and add new version to it -
Python: Copy version subdirectory to
stable/
ordev/
if the version qualifies as the latest version or a development version - Python: Create a sitemap index of individual sitemaps.
-
Python: Download
-
CI:
touch
all files and create a file called.nojekyll
to tell GitHub not to do Jekyll processing -
CI:
git add -A
the version subdirectory -- this means to add changes from all tracked and untracked files.git add
all other relevant files, e.g..nojekyll versions.json *.xml *.html index.html
. -
CI: if any of
dev
,stable
,documentation_pages
exist, add those too. The last one is probably MDAnalysis/mdanalysis only. -
CI: commit and push to gh-pages. The current CI creates a dummy committer called
github-actions
.