ftp.belnet.be mirror is preventing plugin downloads

[RobinBanbury-FR]

Service(s)

infra.ci.jenkins.io, mirrors.jenkins.io, plugins.jenkins.io

Summary

We are being prevented from downloading Jenkins plugins from the ftp.belnet.be mirror

Our Jenkins instances are managed using Puppet. We tried an upgrade today and a lot of the plugins failed to download from ftp.belnet.be/mirror with the message 'Network is unreachable'.

It seems Belnet have recently been using a blocklist - see discussion here: https://groups.google.com/g/jenkins-infra/c/C7cW3MKwR0I

Please could you investigate why we have been blocked?

Output from curl, for reference

$ curl -vvv -L https://updates.jenkins.io/download/plugins/h2-api/1.4.199/h2-api.hpi -o h2-api                                                                                                                                                                                                                                                                                                       23 ✘  1s    13:47:54 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 52.202.51.185:443...
* Connected to [updates.jenkins.io](http://updates.jenkins.io/) (52.202.51.185) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3979 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=[updates.jenkins.io](http://updates.jenkins.io/)
*  start date: Oct 20 05:08:42 2023 GMT
*  expire date: Jan 18 05:08:41 2024 GMT
*  subjectAltName: host "[updates.jenkins.io](http://updates.jenkins.io/)" matched cert's "[updates.jenkins.io](http://updates.jenkins.io/)"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* [HTTP/2] [1] OPENED stream for https://updates.jenkins.io/download/plugins/h2-api/1.4.199/h2-api.hpi
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: [updates.jenkins.io](http://updates.jenkins.io/)]
* [HTTP/2] [1] [:path: /download/plugins/h2-api/1.4.199/h2-api.hpi]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /download/plugins/h2-api/1.4.199/h2-api.hpi HTTP/2
> Host: [updates.jenkins.io](http://updates.jenkins.io/)
> User-Agent: curl/8.4.0
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 302
< date: Thu, 23 Nov 2023 13:48:51 GMT
< server: Apache
< location: https://get.jenkins.io/plugins/h2-api/1.4.199/h2-api.hpi
< content-length: 240
< content-type: text/html; charset=iso-8859-1
<
* Ignoring the response-body
{ [240 bytes data]
100   240  100   240    0     0    858      0 --:--:-- --:--:-- --:--:--   857
* Connection #0 to host [updates.jenkins.io](http://updates.jenkins.io/) left intact
* Issue another request to this URL: 'https://get.jenkins.io/plugins/h2-api/1.4.199/h2-api.hpi'
*   Trying 20.7.178.24:443...
* Connected to [get.jenkins.io](http://get.jenkins.io/) (20.7.178.24) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4039 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=[get.jenkins.io](http://get.jenkins.io/)
*  start date: Nov  4 10:52:28 2023 GMT
*  expire date: Feb  2 10:52:27 2024 GMT
*  subjectAltName: host "[get.jenkins.io](http://get.jenkins.io/)" matched cert's "[get.jenkins.io](http://get.jenkins.io/)"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://get.jenkins.io/plugins/h2-api/1.4.199/h2-api.hpi
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: [get.jenkins.io](http://get.jenkins.io/)]
* [HTTP/2] [1] [:path: /plugins/h2-api/1.4.199/h2-api.hpi]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /plugins/h2-api/1.4.199/h2-api.hpi HTTP/2
> Host: [get.jenkins.io](http://get.jenkins.io/)
> User-Agent: curl/8.4.0
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 302
< date: Thu, 23 Nov 2023 13:48:52 GMT
< content-type: text/html; charset=utf-8
< content-length: 0
< location: https://ftp.belnet.be/mirror/jenkins/plugins/h2-api/1.4.199/h2-api.hpi
< cache-control: private, no-cache
< link: <https://ftp.halifax.rwth-aachen.de/jenkins/plugins/h2-api/1.4.199/h2-api.hpi>; rel=duplicate; pri=1; geo=de
< strict-transport-security: max-age=2592000; includeSubDomains; preload
<
{ [0 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #1 to host [get.jenkins.io](http://get.jenkins.io/) left intact
* Issue another request to this URL: 'https://ftp.belnet.be/mirror/jenkins/plugins/h2-api/1.4.199/h2-api.hpi'
*   Trying 193.190.198.27:443...
*   Trying [2001:6a8:3c80::27]:443...
* Immediate connect fail for 2001:6a8:3c80::27: Network is unreachable

Reproduction steps

Run this curl command:

curl -vvv -L https://updates.jenkins.io/download/plugins/h2-api/1.4.199/h2-api.hpi -o h2-api                                                                                                                                                                                                                                                                                                      

Confirm the actual download site is ftp.belnet.be

If the error is not reproducible, perhaps it's only our IPs that have been blocklisted?

I'm not sure if there's a command to use to ensure the ftp.belnet.be mirror is the one used for the download in order to reproduce this issue.

[dduportal]

Thanks! Email sent to the Belnet admins with your set or public IPs to check with them if the blocking is on their own.

[dduportal]

@RobinBanbury-FR at the same time, we've temporarily disabled the belnet mirror so you should be able to retry your instance rebuilds (you'll be redirected to another mirror).

[dduportal]

• Belnet admins answered and confirmed there are no firewall rules blocking @RobinBanbury-FR public IPs.
• No answer from @RobinBanbury-FR for a traceroute output to check where are packet blocked. But they confirmed they were able to rebuild their Jenkins instance.
  -As such, we are re-enabling FTP belnet mirror