mirrors.jenkins.io redirect incorrect for 2.440 Debian installer

[MarkEWaite]

Service(s)

mirrors.jenkins.io

Summary

The acceptance test that installs the most recent weekly release is failing for Jenkins 2.440

Reproduction steps

1. https://ci.jenkins.io/job/Infra/job/acceptance-tests/job/install-latest-debian-package/lastBuild/console includes the text:

22:50:14  The following NEW packages will be installed:
22:50:14    jenkins net-tools
22:50:14  0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
22:50:14  Need to get 86.1 MB of archives.
22:50:14  After this operation, 87.6 MB of additional disk space will be used.
22:50:14  Get:1 http://deb.debian.org/debian bookworm/main amd64 net-tools amd64 2.10-0.1 [243 kB]
22:50:14  Err:2 https://pkg.jenkins.io/debian binary/ jenkins 2.440
22:50:14    404  Not Found [IP: 20.7.178.24 443]
22:50:14  Fetched 243 kB in 0s (2299 kB/s)
22:50:14  E: Failed to fetch https://mirrors.jenkins.io/debian/archives.jenkins.iodebian/jenkins_2.440_all.deb  404  Not Found [IP: 20.7.178.24 443]
22:50:14  E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Notice the surprising URL at https://mirrors.jenkins.io/debian/archives.jenkins.iodebian/jenkins_2.440_all.deb

It seems to be mixing mirrors.jenkins.io and archives.jenkins.io into a single URL.

[MarkEWaite]

Same message is displayed when I try to update my Ubuntu 22.04 installation from 2.439 to 2.440. The apt installer reports that same strange URL

[dduportal]

That could be related to #3828 (comment).

Gut feeling: during the package creation, mirrors weren't synchronized yet so the mirror redirector (available through both get.jenkins.io and mirrors.jenkins.io) answered the fallback URL, eg. https://mirrors.jenkins.io/debian/jenkins_2.440_all.deb redirected to https://archives.jenkins.io/debian/jenkins_2.440_all.deb.

[dduportal]

=> Nothing related to mirrors.jenkins.io in the package itself, in the packaging repo or in the release repo (and nothing in the packaging build logs).
=> next step, gotta check on pkg.jenkins.io VM. In any cases, after analysis, we'll re-trigger the packagin job and see if it fixes the issue.

[dduportal]

Update: after a carefull review with @smerle33 we found the following Apache rewrite rule which is in charge of redirecting APT HTTP(s) requests for the debian package and could be the reason of the weird URL:

RewriteRule ^binary/(.*)\.deb$ https://mirrors.jenkins.io/debian/$1.deb [R=302,L]

We cannot reproduce the error now: the installation works as expected => all mirrors (including archives.jenkins.io) have been synchronized so the redirection works as expected now

[dduportal]

Sounds like an async problem caused by archives.jenkins.io not being populated yet when you had the issue:

• https://github.com/jenkins-infra/mirror-scripts/blob/master/sync.sh is called by the release process at the end of the packaging uploads
• Fallback does not seem populated anymore but its comments describes the same behavior as we are facing: https://github.com/jenkins-infra/mirror-scripts/blob/bebd7675616a2073173b241ddf5021d098c5f2b4/populate-fallback.sh#L4-L13
• archives.jenkins.io is "populated" by a remote script execution: https://github.com/jenkins-infra/mirror-scripts/blob/master/populate-archives.sh => gotta check the timestamps

[dduportal]

(edit) this comment is amended, see #3894 (comment)

Bingo!

• https://github.com/jenkins-infra/jenkins-infra/blob/production/dist/profile/templates/archives/mirrorsync.erb is run on archives.jenkins.io but it retrieve data from OSUOSL. It's not yet available during the Core release "packaging" job build so archives.jenkins.io is not able to serve the new deb file.
• Then, release is finished, OSUOSL is set up to "unstage" the new files and start serving them.
• Later, the mirrorsync script is run again on archives.jenkins.io by a crontab (https://github.com/jenkins-infra/jenkins-infra/blob/3b29108d5f538f6d1e24cd941f5ca3f074725f69/dist/profile/manifests/archives.pp#L240C5-L245) every 30 min.

=> the attempt made by @MarkEWaite was in-between, so APT was unable to get anything from archives.jenkins.io.

[dduportal]

There is also issues with the latest URL:

curl --verbose --location https://get.jenkins.io/war/latest/jenkins.war
*   Trying [2603:1030:408:5::15a]:443...
* Connected to get.jenkins.io (2603:1030:408:5::15a) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=get.jenkins.io
*  start date: Jan  8 16:53:00 2024 GMT
*  expire date: Apr  7 16:52:59 2024 GMT
*  subjectAltName: host "get.jenkins.io" matched cert's "get.jenkins.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://get.jenkins.io/war/latest/jenkins.war
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: get.jenkins.io]
* [HTTP/2] [1] [:path: /war/latest/jenkins.war]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET /war/latest/jenkins.war HTTP/2
> Host: get.jenkins.io
> User-Agent: curl/8.4.0
> Accept: */*
> 
< HTTP/2 302 
< date: Wed, 10 Jan 2024 10:28:02 GMT
< content-type: text/html; charset=utf-8
< content-length: 0
< cache-control: private, no-cache
< location: /war/latest/archives.jenkins.iowar/latest/jenkins.war
< strict-transport-security: max-age=2592000; includeSubDomains; preload
< 
* Connection #0 to host get.jenkins.io left intact
* Issue another request to this URL: 'https://get.jenkins.io/war/latest/archives.jenkins.iowar/latest/jenkins.war'
* Found bundle for host: 0x600002320480 [can multiplex]
* Re-using existing connection with host get.jenkins.io
* [HTTP/2] [3] OPENED stream for https://get.jenkins.io/war/latest/archives.jenkins.iowar/latest/jenkins.war
* [HTTP/2] [3] [:method: GET]
* [HTTP/2] [3] [:scheme: https]
* [HTTP/2] [3] [:authority: get.jenkins.io]
* [HTTP/2] [3] [:path: /war/latest/archives.jenkins.iowar/latest/jenkins.war]
* [HTTP/2] [3] [user-agent: curl/8.4.0]
* [HTTP/2] [3] [accept: */*]
> GET /war/latest/archives.jenkins.iowar/latest/jenkins.war HTTP/2
> Host: get.jenkins.io
> User-Agent: curl/8.4.0
> Accept: */*
> 
< HTTP/2 404 
< date: Wed, 10 Jan 2024 10:28:02 GMT
< content-type: text/plain; charset=utf-8
< content-length: 10
< x-content-type-options: nosniff
< strict-transport-security: max-age=2592000; includeSubDomains; preload
< 
Not Found
* Connection #0 to host get.jenkins.io left intact

Related to #3860

[dduportal]

Update:

• The 2 issues are correlated to the change of fallback from OSUOSL to archives.jenkins.io on the mirror redirector service get.jenkins.io (ref. [get.jenkins.io] migrate from mirrorbits to mirrorbits-parent chart #3828 (comment))
• The root cause of the problem is a configuration error in the mirrorbits.conf:
  • Commit https://github.com/jenkins-infra/charts-secrets/commit/d09750202b965ab1f338af4def1560767c8b3a29 introduced a syntax error in this file: the fallback URL must contains both a scheme prefix (missing https:// in our case) and a trailing /.
  • Fixed by https://github.com/jenkins-infra/charts-secrets/commit/a5d92df6e499e95dbaa00d3892a52a828b6ed96b (followed by a jenkins-infra/kubernetes-management execution to deploy the secret changes to production)

TL;DR; the fix is:

Fallbacks:
-  - URL: archives.jenkins.io
+  - URL: https://archives.jenkins.io/
    CountryCode: DE
    ContinentCode: NA

[dduportal]

Bingo!

• https://github.com/jenkins-infra/jenkins-infra/blob/production/dist/profile/templates/archives/mirrorsync.erb is run on archives.jenkins.io but it retrieve data from OSUOSL. It's not yet available during the Core release "packaging" job build so archives.jenkins.io is not able to serve the new deb file.

• Then, release is finished, OSUOSL is set up to "unstage" the new files and start serving them.

• Later, the mirrorsync script is run again on archives.jenkins.io by a crontab (https://github.com/jenkins-infra/jenkins-infra/blob/3b29108d5f538f6d1e24cd941f5ca3f074725f69/dist/profile/manifests/archives.pp#L240C5-L245) every 30 min.

=> the attempt made by @MarkEWaite was in-between, so APT was unable to get anything from archives.jenkins.io.

Amending this comment:

• It's not the 30 minute time windows, but the mirrors async. setup:
  • Until there is a mirror with the content up to date and scanned by the mirrorbits instance running at get.jenkins.io, the mirrorbits fallback is used
  • In this (several minutes long) time window, get.jenkins.io was trying to serve URL from the fallback. But this fallback was misconfigured leading to the "weird" URL in 404 ( e.g. https://mirrors.jenkins.io/debian/archives.jenkins.iodebian/jenkins_2.440_all.deb)
  • Several minutes later, the DEB file is known by mirrors so get.jenkins.io started to redirect to a mirror with a proper URL (instead of the broken fallback) => it explains why the DEB installation was working a few hours after @MarkEWaite attempts.

[dduportal]

This issue is now fixed. I'm going to close it.

Please note that we'll have to check if it is OK during next weekly the 16th of January.

[MarkEWaite]

Confirmed that the acceptance test is now passing.

I also started the packaging test that runs a wide range of operating system installations.