don't let builds on other branches break when one branch is renewed but a package access issue causes a breakage

[rlanzafame]

this happened in march 2025 when gitlab was down (yet again) and sphinx-thebe could not be downloaded. it was only triggered when one branch had a commit and needed to be rebuilt. however, ALL branches broke because sphinx-thebe could not be accessed on any of them. we should somehow make sure that if nothing has happened on a branch, absolutely nothing happens to the website that is there.

here is an example showing that some branches failed with type 2, which was unfortunate, because their build should have existed (i didn't check if the caches were simple expired)

[Tom-van-Woudenberg]

All direct git downloads in the requirements file are not part of the cached environment, so that's why this affects all branches.

Btw, in the past we had other packages on Gitlab too

[Tom-van-Woudenberg]

And yes, cache expires depending on the repository settings. Can be up to 90(?) days?

[Tom-van-Woudenberg]

So @rlanzafame , I don't think this is an issue. Or do you think differently?

[rlanzafame]

All direct git downloads in the requirements file are not part of the cached environment, so that's why this affects all branches.

Btw, in the past we had other packages on Gitlab too

Right, and this might happen again in the future (not necessarily us) and I don't want workflows to break the same way they did last weekend.

So @rlanzafame , I don't think this is an issue. Or do you think differently?

I think it is still an issue. I'll come back to this sometime later after I learn more about the caching to see if I can actually propose a way to address it (if possible).

[rlanzafame]

All direct git downloads in the requirements file are not part of the cached environment, so that's why this affects all branches.

Btw, in the past we had other packages on Gitlab too

Actually it looks like the way that packages that are specified as git repos git+... are checked requires connecting over the internet. It would not happen with the pypi packages. You can see it here with an action from the Manual now that the github sphinx-thebe is reported. This apparently happens every time a build is executed, regardless of cache, so it makes things slower unnecessarily. There has to be a better way to do this in the requirements.txt file

Collecting git+https://github.com/TeachBooks/Sphinx-Thebe (from -r requirements.txt (line 13))
  Cloning https://github.com/TeachBooks/Sphinx-Thebe to /tmp/pip-req-build-n6g9hloi
  Running command git clone --filter=blob:none --quiet https://github.com/TeachBooks/Sphinx-Thebe /tmp/pip-req-build-n6g9hloi
  Resolved https://github.com/TeachBooks/Sphinx-Thebe to commit 94dd8702423894fe2f06417a5cc07948767021bb
  Running command git submodule update --init --recursive -q
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'

[rlanzafame]

@Tom-van-Woudenberg do you remember if there was a reason why the packages were not included in the cache? Perhaps they were too big? I wonder if we can improve this by caching at least some packages (see comment above).

[Tom-van-Woudenberg]

@Tom-van-Woudenberg do you remember if there was a reason why the packages were not included in the cache? Perhaps they were too big? I wonder if we can improve this by caching at least some packages (see comment above).

I don't know, but I don't think size is the issue. We're doing this for the caching:

deploy-book-workflow/.github/workflows/deploy-book.yml

Line 186 in d9b4f00

- if: ${{ steps.cache-html.outputs.cache-hit != 'true' }}

[rlanzafame]

Actually, the cache is not really taken full advantage of. Look at this job, it should do absolutely nothing since i made a newline commit (meaningless) to the readme, but instead it takes 1 m 18 s, and a lot of it is unecessary:

• 27 s cloning the repo (seems unavoidable since the hash is created based on files)
• 12 s most of which is probably installing a git package that is not needed
• 16 s preprocessing a branch for....no reason?
  https://github.com/TeachBooks/manual/actions/runs/14211784621/job/39820207174

[Tom-van-Woudenberg]

Actually, the cache is not really taken full advantage of. Look at this job, it should do absolutely nothing since i made a newline commit (meaningless) to the readme, but instead it takes 1 m 18 s, and a lot of it is unecessary:

• 27 s cloning the repo (seems unavoidable since the hash is created based on files)
• 12 s most of which is probably installing a git package that is not needed
• 16 s preprocessing a branch for....no reason?
  https://github.com/TeachBooks/manual/actions/runs/14211784621/job/39820207174

Probably cache was expired, becuase you're actually building the book.

[rlanzafame]

Probably cache was expired, becuase you're actually building the book.

no in this case venv should be reused because requirements.txt was not changed

[rlanzafame]

@Tom-van-Woudenberg do you remember if there was a reason why the packages were not included in the cache? Perhaps they were too big? I wonder if we can improve this by caching at least some packages (see comment above).

I don't know, but I don't think size is the issue. We're doing this for the caching:

deploy-book-workflow/.github/workflows/deploy-book.yml

Line 186 in d9b4f00

 - if: ${{ steps.cache-html.outputs.cache-hit != 'true' }}

wait, we are caching the environments in requirements.txt!

[Tom-van-Woudenberg]

no in this case venv should be reused because requirements.txt was not changed

Yes, the cached venv was used: https://github.com/TeachBooks/manual/actions/runs/14211784621/job/39820207174#step:7:26

[rlanzafame]

Reassessing, now that the README and comments in the yml file have been updated:

• our TeachBooks Template should avoid including anything from TUD GitLab in the requirements.txt as it fails too often (already done)
• we should at minimum advise our users to be aware that the workflow will break if the URL to a package is unavailable. This is very unlikely on PyPI, but on other platforms (like TUD GL it may happen too often)
• we should avoid using packages specified as git+... as the process of cloning a repo during the build process is very slow
• if it is necessary to specify a package as git+... (i.e., while waiting to publish it on PyPI) we should identify a way to skip this step if a cache can be reused, as in this case the repo is not needed but is still cloned when pip runs its package check