API SSL error with some ipv6 addresses

[tootai]

Hi list,

I use Telegram bot on a Debian12 ipv6 server running librenms and Transport being Telegram. I also wrote a script to send messages to a group. Since those last days I didn't receive any alert from Librenms and I discover that curl fail with error 28 SSL connection timeout when testing from Librenms. An error occurs too after 2 minutes when I run curl from command line like

curl https://api.telegram.org/bot/sendMessage?chat_id=&text=Is%20not,%20functional%20yet
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:02:01 --:--:-- 0
curl: (35) Recv failure: Connection reset by peer

Running curl -4 ... and the job is done. api.telegram.org is resolved in ipv4 and ipv6, nc -v <api.telegram.org ipv6> 443 succeeded as well as a mtr to the IP.

I discovered that with another ipv6 address, from the same server, it's OK ! What's going on here ? The failing ipv6 address being from the IP range 2001:67c:1298::/48.

Thanks for your support

[levlam]

For me api.telegram.org resolves to 2001:67c:4e8:f004::9, which isn't in the mentioned network.

[tootai]

Exactly, but why I didn't get full answer with IPv6 belonging to the above /48 range ?

[levlam]

Why should you receive an answer from the 2001:67c:1298::/48 IP address, if api.telegram.org resolves to another address?

[tootai]

OK, seems I didn't explain clearly: my IP is from 2001:67c:1298::/48 range. My other IP is from 2a01:4f8:210::/48 range. Connecting to api using the IP from this second range is OK but when I use the other IP from the first range I see some traffic with tcpdump, outgoing and incoming, which means both servers are connected and exchanging datas but nothing else, it finish with a timeout in terminal mode or error 28 SSL inside Librenms.

Hope you now understand the problematic ;)

[levlam]

This looks like an Internet connectivity issue, which happens much more often with IPv6 networks. You can try to use tcptraceroute to find the cause of the issue and report it to your Internet provider.

[tootai]

I don't think so: from the same IP range other computer I have the same problem. Also this server is ONLY ipv6 and running 24h/24h (backups, web server, emails aso) so I would not put 1 cent of the connectivity. Server is in a DC with other of our servers, no one is showing a problem. And remember, Librenms is running on it watching our servers in various countries, not any alert about such a failure.

[tootai]

mtr report 17ms avg time with 0% loss for 100 packets sent

[levlam]

mtr uses ICMPv6 packets, so its results are often irrelevant when investigating TCP connectivity issues.

[tootai]

Please, dont't try to explain me how mtr/icmp/... works. Generally we all use ICMP/ICMPv6 to check connectivity. If you feel out of idea, just tell it.

Remember with one IP range it's OK the other one not, both using the same Internet provider till a certain point.

[levlam]

If site works from one network and doesn't work from another, then this is very likely to be a network connectivity issue. The used protocol for data transfer is of utmost importance in the case of such network issues. For api.telegram.org there is no difference from which IPv6 address the request is received.

[tootai]

I opened a case on peering partner. I ran a tcptraceroute6 and got for port 80

root@zone-s:/usr/local/bin# tcptraceroute6 api.telegram.org traceroute to api.telegram.org (2001:67c:4e8:f004::9) from 2001:67c:1298::ff04, port 80, from port 17001, 30 hops max, 60 bytes packets 1 2a01:4f8:120:1104::70:1 (2a01:4f8:120:1104::70:1) 0.103 ms 0.146 ms 0.115 ms 2 fd99:a:b:98:67::1 (fd99:a:b:98:67::1) 3.552 ms 3.555 ms 3.306 ms 3 2a00:6340:1000:2024::2 (2a00:6340:1000:2024::2) 7.566 ms 6.779 ms 6.660 ms 4 2a00:6340:1000:2024::1 (2a00:6340:1000:2024::1) 6.857 ms 6.779 ms 6.559 ms 5 C35362-410.C35362-413.telegram.org (2001:7f8::f259:0:2) 17.739 ms 13.356 ms 15.435 ms 6 2001:67c:4e8:801a:1:9102:1:1 (2001:67c:4e8:801a:1:9102:1:1) 13.292 ms 15.342 ms 13.189 ms 7 2001:67c:4e8:801a:1:801e:1:2 (2001:67c:4e8:801a:1:801e:1:2) 15.329 ms 13.406 ms 13.152 ms 8 2001:67c:4e8:eeee:801e:4:0:2632 (2001:67c:4e8:eeee:801e:4:0:2632) 13.457 ms 13.291 ms 13.314 ms 9 2001:67c:4e8:f004::9 (2001:67c:4e8:f004::9) 12.911 ms [open] 12.881 ms 13.014 ms

and for port 443

root@zone-s:/usr/local/bin# tcptraceroute6 api.telegram.org 443 traceroute to api.telegram.org (2001:67c:4e8:f004::9) from 2001:67c:1298::ff04, port 443, from port 16883, 30 hops max, 60 bytes packets 1 2a01:4f8:120:1104::70:1 (2a01:4f8:120:1104::70:1) 0.119 ms 0.159 ms 0.213 ms 2 fd99:a:b:98:67::1 (fd99:a:b:98:67::1) 3.155 ms 3.157 ms 2.949 ms 3 2a00:6340:1000:2024::2 (2a00:6340:1000:2024::2) 6.859 ms 6.714 ms 6.757 ms 4 * * * 5 * * * 6 C35362-410.C35362-413.telegram.org (2001:7f8::f259:0:2) 13.879 ms 14.173 ms 13.704 ms 7 2001:67c:4e8:801a:1:9102:1:1 (2001:67c:4e8:801a:1:9102:1:1) 13.683 ms 13.744 ms 13.570 ms 8 2001:67c:4e8:801a:1:801e:1:2 (2001:67c:4e8:801a:1:801e:1:2) 13.531 ms 13.381 ms 13.570 ms 9 2001:67c:4e8:eeee:801e:4:0:2632 (2001:67c:4e8:eeee:801e:4:0:2632) 13.501 ms 13.552 ms 13.301 ms 10 2001:67c:4e8:f004::9 (2001:67c:4e8:f004::9) 12.885 ms [open] 12.742 ms 12.926 ms

In verbose mode
`* Host api.telegram.org:443 was resolved.

• IPv6: 2001:67c:4e8:f004::9
• IPv4: 149.154.167.220
• Trying [2001:67c:4e8:f004::9]:443...
• Connected to api.telegram.org (2001:67c:4e8:f004::9) port 443
• ALPN: curl offers h2,http/1.1
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• CAfile: /etc/ssl/certs/ca-certificates.crt
• CApath: /etc/ssl/certs
• Recv failure: Connection reset by peer
• OpenSSL SSL_connect: Connection reset by peer in connection to api.telegram.org:443
• Closing connection
  curl: (35) Recv failure: Connection reset by peer`

Nothing more

So like icmpv6, tcp6 seems OK but still in error. Banging my head :)

[levlam]

Could you also try mtr --tcp -P 443?

[levlam]

Could you also try from some other IP address from the 2001:67c:1298::0/48 network other than 2001:67c:1298::ff04?

[tootai]

Same with all 2001:67c:1298::/64 range (I use this /64 and 2001:67c:1298:10::/64 from this /48 range)

root@zone-s:/tmp# mtr --tcp -P 443 --report-wide api.telegram.org
Start: 2024-03-20T15:39:21+0100 [4/1258]
HOST: zone-s Loss% Snt Last Avg Best Wrst StDev
1.|-- 2a01:4f8:120:1104::70:1 0.0% 10 0.4 0.3 0.2 0.5 0.1
2.|-- fd99:a:b:98:67::1 0.0% 10 3.2 3.4 3.2 4.8 0.5
3.|-- 2a00:6340:1000:2024::2 0.0% 10 6.8 7.0 6.7 7.7 0.3
4.|-- 2a00:6340:1000:2024::1 50.0% 10 7.2 7.5 7.2 8.3 0.5
5.|-- C35362-410.C35362-413.telegram.org 60.0% 10 14.5 17.6 13.7 27.9 6.9
6.|-- C35362-410.C35362-413.telegram.org 0.0% 10 15.2 14.7 13.6 19.2 1.7
2001:67c:4e8:801a:1:9102:1:1
2001:67c:4e8:7003:1:9102:1:1
2001:67c:4e8:801b:1:9102:1:1
7.|-- 2001:67c:4e8:801a:1:801e:1:2 0.0% 10 14.3 14.0 13.7 14.3 0.2
2001:67c:4e8:801a:1:9102:1:1
2001:67c:4e8:7003:1:9102:1:1
2001:67c:4e8:801b:1:801f:1:2
2001:67c:4e8:801a:1:801f:1:2
2001:67c:4e8:7003:1:801e:1:2
2001:67c:4e8:801b:1:9102:1:1
8.|-- 2001:67c:4e8:eeee:801f:4:0:2631 0.0% 10 13.8 13.8 13.5 14.1 0.2
2001:67c:4e8:eeee:801e:4:0:2632
2001:67c:4e8:7003:1:801e:1:2
2001:67c:4e8:eeee:801e:4:0:2634
2001:67c:4e8:eeee:801f:4:0:2634
2001:67c:4e8:801a:1:801f:1:2
2001:67c:4e8:eeee:801f:4:0:2633
2001:67c:4e8:801b:1:801e:1:2
2001:67c:4e8:801b:1:801f:1:2
9.|-- 2001:67c:4e8:eeee:801f:4:0:2633 0.0% 10 13.8 13.8 13.4 14.1 0.2
2001:67c:4e8:eeee:801f:4:0:2635
2001:67c:4e8:eeee:801e:4:0:2636
2001:67c:4e8:eeee:801e:4:0:2631
2001:67c:4e8:eeee:801f:4:0:2631
2001:67c:4e8:eeee:801e:4:0:2634
2001:67c:4e8:f004::9
2001:67c:4e8:eeee:801f:4:0:2634
10.|-- 2001:67c:4e8:f004::9 0.0% 8 13.7 13.6 13.3 14.2 0.3

I downloaded with curl from various repos including debian-netinstall.iso, everything like expected.

[tootai]

First packets output of tcpdump from failing IP:

11:33:41.430289` ens1 Out IP6 2001:67c:1298::ff04.60280 > 2001:67c:4e8:f004::9.https: Flags [S], seq 2294335465, win 64800, options [mss 1440,sackOK,TS val 2421837041 ecr 0,nop,wscale 7], length 0

11:33:41.443297 ens1 In IP6 2001:67c:4e8:f004::9.https > 2001:67c:1298::ff04.60280: Flags [S.], seq 3879809308, ack 2294335466, win 28560, options [mss 1440,sackOK,TS val 3435357583 ecr 2421837041,nop,wscale 10], length 0
11:33:41.443327 ens1 Out IP6 2001:67c:1298::ff04.60280 > 2001:67c:4e8:f004::9.https: Flags [.], ack 1, win 507, options [nop,nop,TS val 2421837054 ecr 3435357583], length 0
11:33:41.446243 ens1 Out IP6 2001:67c:1298::ff04.60280 > 2001:67c:4e8:f004::9.https: Flags [P.], seq 1:518, ack 1, win 507, options [nop,nop,TS val 2421837057 ecr 3435357583], length 517
11:33:41.459314 ens1 In IP6 2001:67c:4e8:f004::9.https > 2001:67c:1298::ff04.60280: Flags [.], ack 518, win 29, options [nop,nop,TS val 3435357587 ecr 2421837057], length 0
11:33:41.495913 ens1 In IP6 2001:67c:4e8:f004::9.https > 2001:67c:1298::ff04.60280: Flags [P.], seq 5525:5698, ack 518, win 29, options [nop,nop,TS val 3435357597 ecr 2421837057], length 173
11:33:41.495932 ens1 Out IP6 2001:67c:1298::ff04.60280 > 2001:67c:4e8:f004::9.https: Flags [.], ack 1, win 507, options [nop,nop,TS val 2421837107 ecr 3435357587,nop,nop,sack 1 {5525:5698}], length 0

As you can notice, the before last line is a seq:5525:5698, ack 518 followed by a new ack1. Now the output of a working IP:

11:43:41.211636 ens1 Out IP6 2a01:4f8:120:1104::70:12.49804 > 2001:67c:4e8:f004::9.443: Flags [S], seq 3848103319, win 64800, options [mss 1440,sackOK,TS val 2764072909 ecr 0,nop,wscale 7], length 0
11:43:41.222328 ens1 In IP6 2001:67c:4e8:f004::9.443 > 2a01:4f8:120:1104::70:12.49804: Flags [S.], seq 3343337882, ack 3848103320, win 28560, options [mss 1440,sackOK,TS val 4200978678 ecr 2764072909,nop,wscale 10], length 0
11:43:41.222377 ens1 Out IP6 2a01:4f8:120:1104::70:12.49804 > 2001:67c:4e8:f004::9.443: Flags [.], ack 1, win 507, options [nop,nop,TS val 2764072919 ecr 4200978678], length 0
11:43:41.225508 ens1 Out IP6 2a01:4f8:120:1104::70:12.49804 > 2001:67c:4e8:f004::9.443: Flags [P.], seq 1:518, ack 1, win 507, options [nop,nop,TS val 2764072922 ecr 4200978678], length 517
11:43:41.236092 ens1 In IP6 2001:67c:4e8:f004::9.443 > 2a01:4f8:120:1104::70:12.49804: Flags [.], ack 518, win 29, options [nop,nop,TS val 4200978681 ecr 2764072922], length 0
11:43:41.237072 ens1 In IP6 2001:67c:4e8:f004::9.443 > 2a01:4f8:120:1104::70:12.49804: Flags [.], seq 1:1429, ack 518, win 29, options [nop,nop,TS val 4200978682 ecr 2764072922], length 1428
11:43:41.237082 ens1 Out IP6 2a01:4f8:120:1104::70:12.49804 > 2001:67c:4e8:f004::9.443: Flags [.], ack 1429, win 501, options [nop,nop,TS val 2764072934 ecr 4200978682], length 0

Here the before last line is seq1:1429 followed by the ack 1429 line which does the sequences continuing as they should.

Question now is why in the first case I receive a false seq number with P flag ?

[levlam]

Thank you for the additional information.

Could you run curl --resolve api.telegram.org:443:[2001:67c:4e8:f004::8] https://api.telegram.org/bot123456789:AAAAAAAA/getme, collect a pcap file for the request using tcpdump, and send the file to https://t.me/tdlib_bot?

[tootai]

root@zone-s:~# curl --resolve api.telegram.org:443:[2001:67c:4e8:f004::8] https://api.telegram.org/bot123456789:AAAAAAAA/getme {"ok":false,"error_code":401,"description":"Unauthorized"}root@zone-s:~#

[levlam]

It's really strange that you are able to receive the answer from api.telegram.org through another IPv6 address, but not through the default one. This could mean that someone in the middle intentionally blocks HTTPS to the specififc IPv6 address. Also, it can be seen from tcpdump that TCP connection was established and is dropped some time after it was established, hence this isn't a network connectivity issue.

[tootai]

Do you really meant ipv6 2001:67c:4e8:f004::8 and not ::9 ?

Output in verbose mode (tried with both IPs)

• Added api.telegram.org:443:[2001:67c:4e8:f004::8] to DNS cache
• Hostname api.telegram.org was found in DNS cache
• Trying [2001:67c:4e8:f004::8]:443...
• Connected to api.telegram.org (2001:67c:4e8:f004::8) port 443
• ALPN: curl offers h2,http/1.1
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• CAfile: /etc/ssl/certs/ca-certificates.crt
• CApath: /etc/ssl/certs
• TLSv1.3 (IN), TLS handshake, Server hello (2):
• TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
• TLSv1.3 (IN), TLS handshake, Certificate (11):
• TLSv1.3 (IN), TLS handshake, CERT verify (15):
• TLSv1.3 (IN), TLS handshake, Finished (20):
• TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
• TLSv1.3 (OUT), TLS handshake, Finished (20):
• SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
• ALPN: server accepted h2
• Server certificate:
• subject: CN=api.telegram.org
• start date: Mar 26 07:39:18 2023 GMT
• expire date: Apr 26 07:39:18 2024 GMT
• subjectAltName: host "api.telegram.org" matched cert's "api.telegram.org"
• issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2
• SSL certificate verify ok.
• Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
• Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
• Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
• using HTTP/2
• [HTTP/2] [1] OPENED stream for https://api.telegram.org/bot123456789:AAAAAAAA/getme
• [HTTP/2] [1] [:method: GET]
• [HTTP/2] [1] [:scheme: https]
• [HTTP/2] [1] [:authority: api.telegram.org]
• [HTTP/2] [1] [:path: /bot123456789:AAAAAAAA/getme]
• [HTTP/2] [1] [user-agent: curl/8.6.0]
• [HTTP/2] [1] [accept: /]

GET /bot123456789:AAAAAAAA/getme HTTP/2
Host: api.telegram.org
User-Agent: curl/8.6.0
Accept: /

• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• old SSL session ID is stale, removing
  < HTTP/2 401
  < server: nginx/1.18.0
  < date: Thu, 21 Mar 2024 18:29:35 GMT
  < content-type: application/json
  < content-length: 58
  < strict-transport-security: max-age=31536000; includeSubDomains; preload
  < access-control-allow-origin: *
  < access-control-expose-headers: Content-Length,Content-Type,Date,Server,Connection
  <
• Connection #0 to host api.telegram.org left intact
  {"ok":false,"error_code":401,"description":"Unauthorized"}

[tootai]

I have to add that against ipv6 ending with ::8 I get the same result -above one- with both IPs. If I do the test against ::9, only the "good" ipv6 shows the same result, the failing one giving

curl -v --resolve api.telegram.org:443:[2001:67c:4e8:f004::9] https://api.telegram.org/bot123456789:AAAAAAAA/getme

• Added api.telegram.org:443:[2001:67c:4e8:f004::9] to DNS cache
• Hostname api.telegram.org was found in DNS cache
• Trying [2001:67c:4e8:f004::9]:443...
• Connected to api.telegram.org (2001:67c:4e8:f004::9) port 443
• ALPN: curl offers h2,http/1.1
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• CAfile: /etc/ssl/certs/ca-certificates.crt
• CApath: /etc/ssl/certs

That's all. To summarize:

curl --resolve api.telegram.org:443:[2001:67c:4e8:f004::8] https://api.telegram.org/bot123456789:AAAAAAAA/getme {"ok":false,"error_code":401,"description":"Unauthorized"}

Gives the same result with both IPs as

curl --resolve api.telegram.org:443:[2001:67c:4e8:f004::9] https://api.telegram.org/bot123456789:AAAAAAAA/getme {"ok":false,"error_code":401,"description":"Unauthorized"}

fail on IP src 2001:67c:1298::ff04

[levlam]

Do you really meant ipv6 2001:67c:4e8:f004::8 and not ::9 ?

Yes. We wanted to test with the same server with a different IPv6 address.

[tootai]

So as I get same result with my both ipv6 IPs against ::8 it means that something is wrong with ::9

[tootai]

Up ?

[levlam]

Did you receive response from the peering partner?

[tootai]

They say they are not involved, which seems to be right for me. Connecting to ::8 is OK to ::9 not, so they are in the pass on both IPs and don't fail with ::8

[levlam]

::8 has connected you with a server that is available with ::9, so there are no issues with TCP, IPv6, or Telegram servers. TCP with ::9 also works, so there is a specific issue with using HTTPS from your server to ::9.

[Commifreak]

I have the same issue for the telegram API when using IPv6! Sometimes its just working but most of time its not. v4 always fine.

curl -6 -vvv -S -s -q -X POST "https://api.telegram.org/bot${TOKEN}/sendMessage"
*   Trying 2001:67c:4e8:f004::9:443...
* Connected to api.telegram.org (2001:67c:4e8:f004::9) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to api.telegram.org:443
* Closing connection 0
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to api.telegram.org:443

Any other IPv6 server with curl tests just fine and always works.

[around84]

I fully confirm that there is still a bug:

curl -6 -vvv -S -s -q -X POST "https://api.telegram.org/bot${TOKEN}/sendMessage"
*   Trying [2001:67c:4e8:f004::9]:443...
* Connected to api.telegram.org (2001:67c:4e8:f004::9) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* Recv failure: Соединение разорвано другой стороной
* OpenSSL SSL_connect: Соединение разорвано другой стороной in connection to api.telegram.org:443
* Closing connection 0
curl: (35) Recv failure: Соединение разорвано другой стороной

[levlam]

@around84 This is still looks like an Internet connectivity issue, which happens much more often with IPv6 networks, or a block by government/Internet provider.

[psykokwak-com]

Got this issue too.

[xniksysx]

What if try to set MTU lower then 1500 on ipv6 interface?

[xniksysx]

if i dont set in Mikrotik ipv6 ND MTU then i cant connect to any in *.telegram.org. Check on 2 ISP in Russia. ND MTU set as PPPoE MTU.

[tootai]

No changes on my side. Link to pcap capture https://tdrive.tootai.net/s/GNGzJWr7NQwdkMX