Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent rate limiting on docs build #1822

Open
trevorcampbell opened this issue Aug 28, 2023 · 3 comments
Open

Intermittent rate limiting on docs build #1822

trevorcampbell opened this issue Aug 28, 2023 · 3 comments
Labels
maintenance

Comments

@trevorcampbell
Copy link
Contributor

trevorcampbell commented Aug 28, 2023 •
edited
Loading

When building the docs (most recently the CI step below, which prompted me to file this issue, but other architectures cause the same issue sporadically):

Test python-doc / test_nbgrader (ubuntu-22.04, docs, 3.11) (pull_request)

I have noticed that the workflow is making many quick http requests and is occasionally getting rate limited by GitHub. For example, see https://github.com/jupyter/nbgrader/actions/runs/6005710823/job/16288942885?pr=1817 :

(user_guide/what_is_nbgrader: line  418) ok        https://noteable.edina.ac.uk/documentation/nbguide/
(user_guide/philosophy: line    5) ok        https://www.dropbox.com/s/olrx40rzzvk1v1i/Marr%20-%20The%20Philosophy%20and%20the%20Approach.pdf?dl=0
(contributor_guide/documentation: line   38) ok        https://www.sphinx-doc.org/en/master/
(       changelog: line   85) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Afranswel+updated%3A2023-03-28..2023-06-15&type=Issues
(       changelog: line   85) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Agithub-actions+updated%3A2023-03-28..2023-06-15&type=Issues
(       changelog: line   58) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Agithub-actions+updated%3A2023-06-15..2023-06-16&type=Issues
(       changelog: line   22) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Abrichet+updated%3A2023-07-18..2023-07-21&type=Issues
(       changelog: line  147) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Agithub-actions+updated%3A2022-08-22..2022-09-26&type=Issues
(       changelog: line   38) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Agithub-actions+updated%3A2023-06-16..2023-07-18&type=Issues
(       changelog: line  133) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Adependabot+updated%3A2022-09-27..2023-03-28&type=Issues
(       changelog: line   38) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Abrichet+updated%3A2023-06-16..2023-07-18&type=Issues
(       changelog: line   22) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Agithub-actions+updated%3A2023-07-18..2023-07-21&type=Issues
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajhamrick+updated%3A2022-05-07..2022-06-28&type=Issues | sleeping...
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajhamrick+updated%3A2022-05-07..2022-08-22&type=Issues | sleeping...
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2022-05-07..2022-08-22&type=Issues | sleeping...
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2022-05-07..2022-06-28&type=Issues | sleeping...
(       changelog: line  133) broken    https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2022-09-27..2023-03-28&type=Issues - 429 Client Error: Too Many Requests for url: https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2022-09-27..2023-03-28&type=Issues
(       changelog: line  133) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Agithub-actions+updated%3A2022-09-27..2023-03-28&type=Issues
(       changelog: line  267) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2022-05-07..2022-06-28&type=Issues
(       changelog: line  196) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajhamrick+updated%3A2022-05-07..2022-08-22&type=Issues
(       changelog: line  267) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajhamrick+updated%3A2022-05-07..2022-06-28&type=Issues
(       changelog: line  196) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2022-05-07..2022-08-22&type=Issues
(       changelog: line   38) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ajtpio+updated%3A2023-06-16..2023-07-18&type=Issues
(       changelog: line  133) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Ameeseeksmachine+updated%3A2022-09-27..2023-03-28&type=Issues
(       changelog: line  267) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Aminrk+updated%3A2022-05-07..2022-06-28&type=Issues
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Aperllaghu+updated%3A2022-05-07..2022-06-28&type=Issues | sleeping...
(       changelog: line  196) ok        https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Aminrk+updated%3A2022-05-07..2022-08-22&type=Issues
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Aperllaghu+updated%3A2022-05-07..2022-08-22&type=Issues | sleeping...
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Aperllaghu+updated%3A2022-09-27..2023-03-28&type=Issues | sleeping...
-rate limited-   https://github.com/search?q=repo%3Ajupyter%2Fnbgrader+involves%3Arkdarst+updated%3A2022-05-07..2022-08-22&type=Issues | sleeping...
Error: The action has timed out.

This causes CI to look like it's failing, even when nothing is wrong with the build itself. I believe the fix may just be to slow down these requests a bit, but not sure where that fix should be made and what an appropriate rate is.

The Error: The action has timed out. may actually be GitHub killing the action. It looks as though the sphinx-build -b linkcheck command in the docs Makefile is at least aware of rate-limiting and maybe (?) is exponentially backing off ( see sphinx-doc/sphinx#7388 ) , but perhaps GitHub sees the action sleeping and decides to kill it.

Another potentially useful observation: it seems that the failure occurs shortly after 429 Client Error: Too Many Requests for url: ... -- so perhaps it is indeed that sphinx-build isn't backing off quickly enough...
@trevorcampbell trevorcampbell mentioned this issue Aug 28, 2023
Merged
@brichet
Copy link
Contributor

brichet commented Aug 29, 2023 •
edited
Loading

Thanks @trevorcampbell for opening this issue.
The timeout for the documentation is set to 15 minutes in the github workflow:

nbgrader/.github/workflows/test-docs-python.yml

Line 51 in 9bc0a14

echo "TIMEOUT_MINUTES=15" >> $GITHUB_ENV

Yesterday, a few changes concerning the documentation were merged (#1819). I don't know if this improves the duration but the test following the changes took less than 12 minutes: https://github.com/jupyter/nbgrader/actions/runs/6004868762/job/16286405021#logs

@trevorcampbell
Copy link
Contributor Author

@brichet Since you merged #1819 I haven't seen the issue again. But it was somewhat intermittent before, so hard to tell if it's actually gone.

@trevorcampbell
Copy link
Contributor Author

Hah. 10 minutes after I wrote that, it happened again.

https://github.com/jupyter/nbgrader/actions/runs/6018862753/job/16327800386?pr=1817

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
maintenance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@trevorcampbell @brichet