connect from behind corporate proxy

[CyberSeppi]

Hi,

I have a strange problem when connecting from behind corp proxy.
What I´m trying to do is easy:
aws s3 ls --profile my_profile

Here is my setup

Client, from which I´m trying to connect:
Ubuntu@WSL-2@Windows10
proxy environment is set, curl, wget, etc. works as expected

WSL-Host is Windows10 and here runs px.exe (proxy which abstracts auth part and takes info from windows credential store)

What happens

Client side:

Traceback (most recent call last):
  File "awscli/botocore/httpsession.py", line 448, in send
  File "urllib3/connectionpool.py", line 787, in urlopen
  File "urllib3/util/retry.py", line 525, in increment
  File "urllib3/packages/six.py", line 769, in reraise
  File "urllib3/connectionpool.py", line 700, in urlopen
  File "urllib3/connectionpool.py", line 996, in _prepare_proxy
  File "urllib3/connection.py", line 374, in connect
  File "http/client.py", line 924, in _tunnel
urllib3.exceptions.ProxyError: ('Cannot connect to proxy.', OSError('Tunnel connection failed: 503 Service Unavailable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "awscli/botocore/endpoint.py", line 199, in _do_get_response

On Host side (px debug log)

Process-1: Thread_1: 1679477124: do_transaction: No auth required
Process-1: Thread_1: 1679477124: do_CONNECT: Error 503
Process-1: Thread_1: 1679477124: log_message: code 503, message Service Unavailable
Process-1: Thread_1: 1679477124: log_message: "CONNECT sts.eu-central-4.amazonaws.com:443 HTTP/1.0" 503 -
Process-1: Thread_1: 1679477125: do_CONNECT: Cleanup proxy connection
Process-1: Thread_1: 1679477125: do_CONNECT: 0 bytes read, 0 bytes written
Process-1: Thread_1: 1679477125: do_CONNECT: Done

curl on wsl side

curl http://s3.amazonaws.com -vvv
* Uses proxy env variable no_proxy == 'localhost,127.0.0.1,wsl.host,169.254.254.1'
* Uses proxy env variable http_proxy == 'http://169.254.254.1:3128/'
*   Trying 169.254.254.1:3128...
* TCP_NODELAY set
* Connected to 169.254.254.1 (169.254.254.1) port 3128 (#0)
> GET http://s3.amazonaws.com/ HTTP/1.1
> Host: s3.amazonaws.com
> User-Agent: curl/7.68.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 307 Temporary Redirect
< Server: SimpleHTTP/0.6 Python/3.7.5
< Date: Wed, 22 Mar 2023 09:27:12 GMT
< x-amz-id-2: <censored>
< x-amz-request-id: PF5X9FHVQJ8JX0DB
< Date: Wed, 22 Mar 2023 09:27:13 GMT
< Location: https://aws.amazon.com/s3/
< Server: AmazonS3
< Content-Length: 0
< Proxy-Connection: Keep-Alive
< Connection: Keep-Alive
< X-BlueCoat-Authorization: 8be9fc04xxxxxb2=326206C1000000xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxe0QAsAQAAAAAAALJtlAHHHHHHHHH
< 
* Connection #0 to host 169.254.254.1 left intact

but

curl https://sts.eu-central-4.amazonaws.com -vvv
* Uses proxy env variable no_proxy == 'localhost,127.0.0.1,wsl.host,169.254.254.1'
* Uses proxy env variable https_proxy == 'http://169.254.254.1:3128/'
*   Trying 169.254.254.1:3128...
* TCP_NODELAY set
* Connected to 169.254.254.1 (169.254.254.1) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to sts.eu-central-4.amazonaws.com:443
> CONNECT sts.eu-central-4.amazonaws.com:443 HTTP/1.1
> Host: sts.eu-central-4.amazonaws.com:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 503 Service Unavailable
< Server: SimpleHTTP/0.6 Python/3.7.5
< Date: Wed, 22 Mar 2023 09:31:21 GMT
< Connection: close
< Content-Type: text/html;charset=utf-8
< Content-Length: 484
< 
* Received HTTP code 503 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 503 from proxy after CONNECT

So it seems, that aws-cli is making a call to sts within the whole process and that sems to go awry :-(

Anyone has any tipps for me ?

Thanks guys!

[tim-finnigan]

Here is documentation on using a proxy with the AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-proxy.html. There may be configuration settings on your proxy that are preventing the connection.

[Augustin-FL]

Not exactly what you are looking for, but FYI someone found a way to use boto3 behind corporate proxy with NTLM/Kerberos authentication.

https://github.com/mpieters3/urllib3_kerberos_proxy/blob/master/kerbmonkey/kerb_monkey.py#L55

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.