Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Remove python.sytem_packages from ReadTheDocs config #2267

Merged
merged 1 commit into from
Aug 10, 2023

Conversation

matthewfeickert
Copy link
Member

@matthewfeickert matthewfeickert commented Aug 7, 2023

Description

  • ReadTheDocs is removing the "use system packages" feature on 2023-08-29. The user should fully specify the docs environment, which for pyhf is already done, but pyhf was following the default template which included the uses of system_packages.

The following email was received:

Hello,

You are receving this email because your Read the Docs project is impacted by an upcoming deprecation.

Read the Docs used to pre-install common scientific Python packages like scipy, numpy, pandas, matplotlib and others at system level to speed up the build process. However, with all the work done in the Python ecosystem and the introduction of "wheels", these packages are a lot easier to install via pip install and these pre-installed packages are not required anymore. If you have Apt package dependencies, they can be installed with build.apt_packages.

With the introduction of our new "Ubuntu 20.04" and "Ubuntu 22.04" Docker images, we stopped pre-installing these extra Python packages and we encouraged users to install and pin all their dependencies using a requirements.txt file. We have already stopped supporting "use system packages" on these newer images.

We are removing the "use system packages" feature on August 29th. Make sure you are installing all the required dependecies to build your project's documentation using a requirements.txt file and specifying it in your .readthedocs.yaml.

Here you have an example of the section required on the .readthedocs.yaml configuration file:

python:
  install:
    - requirements: docs/requirements.txt

The content of docs/requirements.txt would be similar to:

scipy==1.11.1
numpy==1.25.2
pandas==2.0.3
matplotlib==3.7.2

We are sending you this email because you are a maintainer of the following projects that are affected by this removal. Either using "Use system package" checkbox in the Admin UI, or the config key python.sytem_packages or python.use_system_site_packages in your .readthedocs.yaml file:

pyhf

Read more about this in our Reproducible builds guide for more details.

Keep documenting,
Read the Docs

Checklist Before Requesting Reviewer

  • Tests are passing
  • "WIP" removed from the title of the pull request
  • Selected an Assignee for the PR to be responsible for the log summary

Before Merging

For the PR Assignees:

  • Summarize commit messages into a comprehensive review of the PR
* ReadTheDocs is removing the "use system packages" feature on
  2023-08-29. The user should fully specify the docs environment, which
  for pyhf is already done, but pyhf was following the default template
  which included the uses of system_packages.

@matthewfeickert matthewfeickert added docs Documentation related need-to-backport tmp label until can be backported to patch release branch labels Aug 7, 2023
@matthewfeickert matthewfeickert self-assigned this Aug 7, 2023
@matthewfeickert
Copy link
Member Author

@kratsg FYI I assume you're going to get hit by this as well.

@codecov
Copy link

codecov bot commented Aug 7, 2023

Codecov Report

Patch and project coverage have no change.

Comparison is base (998ba7c) 98.30% compared to head (8589fe9) 98.30%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2267   +/-   ##
=======================================
  Coverage   98.30%   98.30%           
=======================================
  Files          69       69           
  Lines        4534     4534           
  Branches      802      802           
=======================================
  Hits         4457     4457           
  Misses         45       45           
  Partials       32       32           
Flag Coverage Δ
contrib 97.88% <ø> (ø)
doctest 61.09% <ø> (ø)
unittests-3.10 96.31% <ø> (?)
unittests-3.11 96.31% <ø> (ø)
unittests-3.8 96.33% <ø> (ø)
unittests-3.9 96.36% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@matthewfeickert
Copy link
Member Author

Failing on RTD with:

writing output... 
building [html]: targets for 38 source files that are out of date
updating environment: [new config] 160 added, 0 changed, 0 removed
�[2Kreading sources... [  1%] _generated/pyhf.compat
�[2Kreading sources... [  1%] _generated/pyhf.compat.interpret_rootname
�[2Kreading sources... [  2%] _generated/pyhf.compat.paramset_to_rootnames
�[2Kreading sources... [  2%] _generated/pyhf.contrib.utils.download
�[2Kreading sources... [  3%] _generated/pyhf.contrib.viz.brazil
�[2Kreading sources... [  4%] _generated/pyhf.contrib.viz.brazil.BrazilBandCollection
�[2Kreading sources... [  4%] _generated/pyhf.contrib.viz.brazil.plot_brazil_band
�[2Kreading sources... [  5%] _generated/pyhf.contrib.viz.brazil.plot_cls_components
�[2Kreading sources... [  6%] _generated/pyhf.contrib.viz.brazil.plot_results
�[2Kreading sources... [  6%] _generated/pyhf.exceptions.ImportBackendError
�[2Kreading sources... [  7%] _generated/pyhf.exceptions.InvalidBackend
�[2Kreading sources... [  8%] _generated/pyhf.exceptions.InvalidInterpCode
�[2Kreading sources... [  8%] _generated/pyhf.exceptions.InvalidMeasurement
�[2Kreading sources... [  9%] _generated/pyhf.exceptions.InvalidModel
�[2Kreading sources... [  9%] _generated/pyhf.exceptions.InvalidModifier
�[2Kreading sources... [ 10%] _generated/pyhf.exceptions.InvalidNameReuse
�[2Kreading sources... [ 11%] _generated/pyhf.exceptions.InvalidOptimizer
�[2Kreading sources... [ 11%] _generated/pyhf.exceptions.InvalidPatchLookup
�[2Kreading sources... [ 12%] _generated/pyhf.exceptions.InvalidPatchSet
�[2Kreading sources... [ 12%] _generated/pyhf.exceptions.InvalidPdfData
�[2Kreading sources... [ 13%] _generated/pyhf.exceptions.InvalidPdfParameters
�[2Kreading sources... [ 14%] _generated/pyhf.exceptions.InvalidSpecification
�[2Kreading sources... [ 14%] _generated/pyhf.exceptions.InvalidWorkspaceOperation
�[2Kreading sources... [ 15%] _generated/pyhf.exceptions.PatchSetVerificationError
�[2Kreading sources... [ 16%] _generated/pyhf.get_backend
�[2Kreading sources... [ 16%] _generated/pyhf.infer.calculators.AsymptoticCalculator
�[2Kreading sources... [ 17%] _generated/pyhf.infer.calculators.AsymptoticTestStatDistribution
�[2Kreading sources... [ 18%] _generated/pyhf.infer.calculators.EmpiricalDistribution
�[2Kreading sources... [ 18%] _generated/pyhf.infer.calculators.HypoTestFitResults
�[2Kreading sources... [ 19%] _generated/pyhf.infer.calculators.ToyCalculator
�[2Kreading sources... [ 19%] _generated/pyhf.infer.calculators.generate_asimov_data
�[2Kreading sources... [ 20%] _generated/pyhf.infer.hypotest
�[2Kreading sources... [ 21%] _generated/pyhf.infer.intervals.upper_limits.linear_grid_scan
�[2Kreading sources... [ 21%] _generated/pyhf.infer.intervals.upper_limits.toms748_scan
�[2Kreading sources... [ 22%] _generated/pyhf.infer.intervals.upper_limits.upper_limit
�[2Kreading sources... [ 22%] _generated/pyhf.infer.intervals.upperlimit
�[2Kreading sources... [ 23%] _generated/pyhf.infer.mle.fit
�[2Kreading sources... [ 24%] _generated/pyhf.infer.mle.fixed_poi_fit
�[2Kreading sources... [ 24%] _generated/pyhf.infer.mle.twice_nll
�[2Kreading sources... [ 25%] _generated/pyhf.infer.test_statistics.q0
�[2Kreading sources... [ 26%] _generated/pyhf.infer.test_statistics.qmu
�[2Kreading sources... [ 26%] _generated/pyhf.infer.test_statistics.qmu_tilde
�[2Kreading sources... [ 27%] _generated/pyhf.infer.test_statistics.tmu
�[2Kreading sources... [ 28%] _generated/pyhf.infer.test_statistics.tmu_tilde
�[2Kreading sources... [ 28%] _generated/pyhf.infer.utils.all_pois_floating
�[2Kreading sources... [ 29%] _generated/pyhf.infer.utils.create_calculator
�[2Kreading sources... [ 29%] _generated/pyhf.infer.utils.get_test_stat
�[2Kreading sources... [ 30%] _generated/pyhf.interpolators.code0
�[2Kreading sources... [ 31%] _generated/pyhf.interpolators.code1
�[2Kreading sources... [ 31%] _generated/pyhf.interpolators.code2
�[2Kreading sources... [ 32%] _generated/pyhf.interpolators.code4
�[2Kreading sources... [ 32%] _generated/pyhf.interpolators.code4p
�[2Kreading sources... [ 33%] _generated/pyhf.mixins._ChannelSummaryMixin
�[2Kreading sources... [ 34%] _generated/pyhf.modifiers.histosys
�[2Kreading sources... [ 34%] _generated/pyhf.modifiers.histosys.histosys_builder
�[2Kreading sources... [ 35%] _generated/pyhf.modifiers.histosys.histosys_combined
�[2Kreading sources... [ 36%] _generated/pyhf.modifiers.histosys.required_parset
�[2Kreading sources... [ 36%] _generated/pyhf.modifiers.normfactor
�[2Kreading sources... [ 37%] _generated/pyhf.modifiers.normfactor.normfactor_builder
�[2Kreading sources... [ 38%] _generated/pyhf.modifiers.normfactor.normfactor_combined
�[2Kreading sources... [ 38%] _generated/pyhf.modifiers.normfactor.required_parset
�[2Kreading sources... [ 39%] _generated/pyhf.modifiers.normsys
�[2Kreading sources... [ 39%] _generated/pyhf.modifiers.normsys.normsys_builder
�[2Kreading sources... [ 40%] _generated/pyhf.modifiers.normsys.normsys_combined
�[2Kreading sources... [ 41%] _generated/pyhf.modifiers.normsys.required_parset
�[2Kreading sources... [ 41%] _generated/pyhf.modifiers.shapefactor
�[2Kreading sources... [ 42%] _generated/pyhf.modifiers.shapefactor.required_parset
�[2Kreading sources... [ 42%] _generated/pyhf.modifiers.shapefactor.shapefactor_builder
�[2Kreading sources... [ 43%] _generated/pyhf.modifiers.shapefactor.shapefactor_combined
�[2Kreading sources... [ 44%] _generated/pyhf.modifiers.shapesys
�[2Kreading sources... [ 44%] _generated/pyhf.modifiers.shapesys.required_parset
�[2Kreading sources... [ 45%] _generated/pyhf.modifiers.shapesys.shapesys_builder
�[2Kreading sources... [ 46%] _generated/pyhf.modifiers.shapesys.shapesys_combined
�[2Kreading sources... [ 46%] _generated/pyhf.modifiers.staterror
�[2Kreading sources... [ 47%] _generated/pyhf.modifiers.staterror.required_parset
�[2Kreading sources... [ 48%] _generated/pyhf.modifiers.staterror.staterror_builder
�[2Kreading sources... [ 48%] _generated/pyhf.modifiers.staterror.staterror_combined
�[2Kreading sources... [ 49%] _generated/pyhf.optimize.mixins.OptimizerMixin
�[2Kreading sources... [ 49%] _generated/pyhf.optimize.opt_minuit.minuit_optimizer
�[2Kreading sources... [ 50%] _generated/pyhf.optimize.opt_scipy.scipy_optimizer
�[2Kreading sources... [ 51%] _generated/pyhf.optimizer
�[2Kreading sources... [ 51%] _generated/pyhf.patchset.Patch
�[2Kreading sources... [ 52%] _generated/pyhf.patchset.PatchSet
�[2Kreading sources... [ 52%] _generated/pyhf.pdf.Model
�[2Kreading sources... [ 53%] _generated/pyhf.pdf._ModelConfig
�[2Kreading sources... [ 54%] _generated/pyhf.probability.Independent
�[2Kreading sources... [ 54%] _generated/pyhf.probability.Normal
�[2Kreading sources... [ 55%] _generated/pyhf.probability.Poisson
�[2Kreading sources... [ 56%] _generated/pyhf.probability.Simultaneous
�[2Kreading sources... [ 56%] _generated/pyhf.readxml
�[2Kreading sources... [ 57%] _generated/pyhf.readxml.clear_filecache
�[2Kreading sources... [ 57%] _generated/pyhf.readxml.dedupe_parameters
�[2Kreading sources... [ 58%] _generated/pyhf.readxml.extract_error
�[2Kreading sources... [ 59%] _generated/pyhf.readxml.import_root_histogram
�[2Kreading sources... [ 59%] _generated/pyhf.readxml.parse
�[2Kreading sources... [ 60%] _generated/pyhf.readxml.process_channel
�[2Kreading sources... [ 61%] _generated/pyhf.readxml.process_data
�[2Kreading sources... [ 61%] _generated/pyhf.readxml.process_measurements
�[2Kreading sources... [ 62%] _generated/pyhf.readxml.process_sample
�[2Kreading sources... [ 62%] _generated/pyhf.schema
�[2Kreading sources... [ 63%] _generated/pyhf.schema.Schema
�[2Kreading sources... [ 64%] _generated/pyhf.schema.load_schema
�[2Kreading sources... [ 64%] _generated/pyhf.schema.validate
�[2Kreading sources... [ 65%] _generated/pyhf.set_backend
�[2Kreading sources... [ 66%] _generated/pyhf.simplemodels.correlated_background
�[2Kreading sources... [ 66%] _generated/pyhf.simplemodels.uncorrelated_background
�[2Kreading sources... [ 67%] _generated/pyhf.tensor.jax_backend.jax_backend
�[2Kreading sources... [ 68%] _generated/pyhf.tensor.numpy_backend.numpy_backend
�[2Kreading sources... [ 68%] _generated/pyhf.tensor.pytorch_backend.pytorch_backend
�[2Kreading sources... [ 69%] _generated/pyhf.tensor.tensorflow_backend.tensorflow_backend
�[2Kreading sources... [ 69%] _generated/pyhf.tensorlib
�[2Kreading sources... [ 70%] _generated/pyhf.utils.citation
�[2Kreading sources... [ 71%] _generated/pyhf.utils.digest
�[2Kreading sources... [ 71%] _generated/pyhf.utils.options_from_eqdelimstring
�[2Kreading sources... [ 72%] _generated/pyhf.workspace.Workspace
�[2Kreading sources... [ 72%] _generated/pyhf.writexml
�[2Kreading sources... [ 73%] _generated/pyhf.writexml.build_channel
�[2Kreading sources... [ 74%] _generated/pyhf.writexml.build_data
�[2Kreading sources... [ 74%] _generated/pyhf.writexml.build_measurement
�[2Kreading sources... [ 75%] _generated/pyhf.writexml.build_modifier
�[2Kreading sources... [ 76%] _generated/pyhf.writexml.build_sample
�[2Kreading sources... [ 76%] _generated/pyhf.writexml.indent
�[2Kreading sources... [ 77%] api
�[2Kreading sources... [ 78%] babel
�[2Kreading sources... [ 78%] citations
�[2Kreading sources... [ 79%] cli
�[2Kreading sources... [ 79%] contributors
�[2Kreading sources... [ 80%] development
�[2Kreading sources... [ 81%] examples
�[2Kreading sources... [ 81%] examples/notebooks/ShapeFactor
/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/nbsphinx/__init__.py:1058: RuntimeWarning: You are using an unsupported version of pandoc (2.9.2.1).
Your version must be at least (2.14.2) but less than (4.0.0).
Refer to https://pandoc.org/installing.html.
Continuing with doubts...
  nbconvert.utils.pandoc.check_pandoc_version()
�[2Kreading sources... [ 82%] examples/notebooks/XML_ImportExport
�[2Kreading sources... [ 82%] examples/notebooks/altair
�[2Kreading sources... [ 83%] examples/notebooks/binderexample/StatisticalAnalysis
�[2Kreading sources... [ 84%] examples/notebooks/hello-world
�[2Kreading sources... [ 84%] examples/notebooks/learn/InterpolationCodes
�[2Kreading sources... [ 85%] examples/notebooks/learn/TensorizingInterpolations
�[2Kreading sources... [ 86%] examples/notebooks/learn/TestStatistics
�[2Kreading sources... [ 86%] examples/notebooks/learn/UsingCalculators
�[2Kreading sources... [ 87%] examples/notebooks/multiBinPois
�[2Kreading sources... [ 88%] examples/notebooks/multichannel-coupled-histo
�[2Kreading sources... [ 88%] examples/notebooks/toys
�[2Kreading sources... [ 89%] faq
�[2Kreading sources... [ 89%] governance/ROADMAP
�[2Kreading sources... [ 90%] index
�[2Kreading sources... [ 91%] installation
�[2Kreading sources... [ 91%] intro
�[2Kreading sources... [ 92%] jupyterlite
�[2Kreading sources... [ 92%] learn
�[2Kreading sources... [ 93%] likelihood
�[2Kreading sources... [ 94%] outreach
�[2Kreading sources... [ 94%] release-notes
�[2Kreading sources... [ 95%] release-notes/v0.5.3
�[2Kreading sources... [ 96%] release-notes/v0.5.4
�[2Kreading sources... [ 96%] release-notes/v0.6.0
�[2Kreading sources... [ 97%] release-notes/v0.6.1
�[2Kreading sources... [ 98%] release-notes/v0.6.2
�[2Kreading sources... [ 98%] release-notes/v0.6.3
�[2Kreading sources... [ 99%] release-notes/v0.7.0
�[2Kreading sources... [ 99%] release-notes/v0.7.1
�[2Kreading sources... [100%] release-notes/v0.7.2
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
copying assets... copying static files... done
copying extra files... done
done
�[2Kwriting output... [  1%] _generated/pyhf.compat
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/sphinx/builders/html/__init__.py", line 1098, in handle_page
    output = self.templates.render(templatename, ctx)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/readthedocs_ext/readthedocs.py", line 181, in rtd_render
    content = old_render(template, render_context)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/sphinx/jinja2glue.py", line 196, in render
    return self.environment.get_template(template).render(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/jinja2/environment.py", line 1291, in render
    self.environment.handle_exception()
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/sphinx/themes/basic/page.html", line 10, in top-level template code
    {%- extends "layout.html" %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/sphinx_rtd_theme/layout.html", line 67, in top-level template code
    <link rel="stylesheet" href="{{ pathto('_static/' + style, 1) }}" type="text/css" />
    ^^^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: 'style' is undefined

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/sphinx/cmd/build.py", line 290, in build_main
    app.build(args.force_all, args.filenames)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/sphinx/application.py", line 351, in build
    self.builder.build_update()
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 290, in build_update
    self.build(to_build,
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 360, in build
    self.write(docnames, list(updated_docnames), method)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 567, in write
    self._write_serial(sorted(docnames))
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 577, in _write_serial
    self.write_doc(docname, doctree)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/sphinx/builders/html/__init__.py", line 675, in write_doc
    self.handle_page(docname, ctx, event_arg=doctree)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pyhf/envs/2267/lib/python3.11/site-packages/sphinx/builders/html/__init__.py", line 1105, in handle_page
    raise ThemeError(__("An error happened in rendering the page %s.\nReason: %r") %
sphinx.errors.ThemeError: An error happened in rendering the page _generated/pyhf.compat.
Reason: UndefinedError("'style' is undefined")

Theme error:
An error happened in rendering the page _generated/pyhf.compat.
Reason: UndefinedError("'style' is undefined")

@matthewfeickert
Copy link
Member Author

This should resolve itself with readthedocs/sphinx_rtd_theme#1463. Just need to wait for sphinx-rtd-theme v1.3.0 to get released (c.f. readthedocs/sphinx_rtd_theme#1463 (comment)).

* ReadTheDocs is removing the "use system packages" feature on
  2023-08-29. The user should fully specify the docs environment, which
  for pyhf is already done, but pyhf was following the default template
  which included the uses of system_packages.
@matthewfeickert matthewfeickert force-pushed the docs/remove-python-sytem_packages-rtd branch from e04dee4 to 8589fe9 Compare August 10, 2023 05:20
@matthewfeickert
Copy link
Member Author

I'm going to approve and merge this myself. As always, PRs approved by a single core dev can be reverted as needed by the rest of the dev team.

@matthewfeickert matthewfeickert merged commit 6c35247 into main Aug 10, 2023
19 checks passed
@matthewfeickert matthewfeickert deleted the docs/remove-python-sytem_packages-rtd branch August 10, 2023 07:21
@matthewfeickert matthewfeickert removed the need-to-backport tmp label until can be backported to patch release branch label Aug 16, 2023
matthewfeickert added a commit that referenced this pull request Aug 16, 2023
* Backport components of:
   - PR #2252
   - PR #2255
   - PR #2267
   - PR #2271
   - PR #2272
   - PR #2276
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs Documentation related
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

1 participant