Dynamic forward proxy with connect termination

[shibin-george]

Description:
I am trying to set up Envoy to act as a forward proxy for my internet-outbound needs.
essentially, curl -x <Envoy-service's public IP>:<listener port> xyz.com
should result in Envoy starting a TCP tunnel between client (curl) and destination.

Here's the config that I'm using:

admin:
  address:
    socket_address:
      protocol: TCP
      address: 127.0.0.1
      port_value: 9901
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: egress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  connect_matcher:
                    {}
                route:
                  cluster: dynamic_forward_proxy_cluster
                  upgrade_configs:
                    - upgrade_type: CONNECT
                      connect_config:
                        {}
          http_filters:
          - name: envoy.filters.http.dynamic_forward_proxy
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
              dns_cache_config:
                name: dynamic_forward_proxy_cache_config
                dns_lookup_family: V4_ONLY
                typed_dns_resolver_config:
                  name: envoy.network.dns_resolver.cares
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.network.dns_resolver.cares.v3.CaresDnsResolverConfig
                    resolvers:
                    - socket_address:
                        address: "8.8.8.8"
                        port_value: 53
                    dns_resolver_options:
                      use_tcp_for_dns_lookups: true
                      no_default_search_domain: true
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: dynamic_forward_proxy_cluster
    connect_timeout: 2s
    lb_policy: CLUSTER_PROVIDED
    cluster_type:
      name: envoy.clusters.dynamic_forward_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
        dns_cache_config:
          name: dynamic_forward_proxy_cache_config
          dns_lookup_family: V4_ONLY
          typed_dns_resolver_config:
            name: envoy.network.dns_resolver.cares
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.network.dns_resolver.cares.v3.CaresDnsResolverConfig
              resolvers:
              - socket_address:
                  address: "8.8.8.8"
                  port_value: 53
              dns_resolver_options:
                use_tcp_for_dns_lookups: true
                no_default_search_domain: true

When I run curl -x IP:10010 https://api.ipify.org -v, here's the output I get:

* Connected to <REDACTED> port 10010 (#0)
* allocate connect buffer
* Establish HTTP proxy tunnel to api.ipify.org:443
> CONNECT api.ipify.org:443 HTTP/1.1
> Host: api.ipify.org:443
> User-Agent: curl/7.83.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 404 Not Found
< date: Tue, 22 Nov 2022 06:41:38 GMT
< server: istio-envoy
< connection: close
< content-length: 0
<
* Received HTTP code 404 from proxy after CONNECT
* CONNECT phase completed
* Closing connection 0
curl: (56) Received HTTP code 404 from proxy after CONNECT

If the destination is http instead of https, its still a 404, but a different reason:

`curl -x IP:10010 http://api.ipify.org -v`
* Connected to <REDACTED> port 10010 (#0)
> GET http://api.ipify.org/ HTTP/1.1
> Host: api.ipify.org
> User-Agent: curl/7.83.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< date: Tue, 22 Nov 2022 06:43:30 GMT
< server: istio-envoy
< content-length: 0
< x-envoy-upstream-service-time: 0
< x-envoy-decorator-operation: envoy-forward-proxy.default.svc.cluster.local:10010/*
<

My config is borrowed from here but for DFP cluster type.
Appreciate any help in figuring out what's going wrong here.
squid as forward proxy is working but I don't want to use squid, if I can.

[shibin-george]

@batchamalick I noticed that you posted a config here that terminated CONNECT and creates tunnel to destination. I amt rying basically the same thing but all I get is 404s for some reason.
Any ideas on where am I going wrong? Thanks!

[batchamalick]

is the port 10010 is configured as in your config the port is 10000 ?

[shibin-george]

is the port 10010 is configured as in your config the port is 10000 ?

Oh yeah, 10010 is the kubernetes service's port that exposes the envoy deployment. My envoy port is 10000, and the service's port 10010 gets translated to 10000

[batchamalick]

I'm not sure what envoy you are running but with the same config you shared I can do the test.

curl -x 127.0.0.1:10000 https://api.ipify.org -v
*   Trying 127.0.0.1:10000...
* Connected to 127.0.0.1 (127.0.0.1) port 10000 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to api.ipify.org:443
> CONNECT api.ipify.org:443 HTTP/1.1
> Host: api.ipify.org:443
> User-Agent: curl/7.79.1
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK

My envoy version

envoy --version

envoy  version: edd69583372955fdfa0b8ca3820dd7312c094e46/1.23.1/Modified/RELEASE/BoringSSL

[wbpcode]

cc @alyssawilk

[shibin-george]

alright, so this is tricky and I don't quite know how to explain this. Here's what is happening:

• I deploy envoy forward-proxy (with config same as what I mentioned at the top) in a kubernetes cluster.
• Then, I expose the deployment as a Kubernetes service.
  • kubectl expose deployment envoy-forward-proxy --type=LoadBalancer --name=envoy-forward-proxy --port=10010 --target-port=10000
  • This gives me a public IP address for the envoy-forward-proxy.
• Now:
  • curl -x <envoy public IP>:10010 https://api.ipify.org -v
    • I get a 404
  • curl -x <envoy cluster IP>:10010 https://api.ipify.org -v
    • also a 404
  • If I perform curl from the istio-proxy sidecar container alongside the envoy container (the istio-proxy sidecar has curl)
    • curl -x 127.0.0.1:10000 https://api.ipify.org -v
    • This time, its a 200. Output:

istio-proxy@envoy-forward-proxy-7d68f66c6-ztkgn:/$ curl -x 127.0.0.1:10000 https://api.ipify.org -v
*   Trying 127.0.0.1:10000...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 10000 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to api.ipify.org:443
> CONNECT api.ipify.org:443 HTTP/1.1
> Host: api.ipify.org:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< date: Thu, 24 Nov 2022 22:45:30 GMT
< server: envoy
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.ipify.org
*  start date: Feb  7 00:00:00 2022 GMT
*  expire date: Mar 10 23:59:59 2023 GMT
*  subjectAltName: host "api.ipify.org" matched cert's "*.ipify.org"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: api.ipify.org
> User-Agent: curl/7.68.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: Cowboy
< Connection: keep-alive
< Content-Type: text/plain
< Vary: Origin
< Date: Thu, 24 Nov 2022 22:45:31 GMT
< Content-Length: 14
< Via: 1.1 vegur
<
* Connection #0 to host 127.0.0.1 left intact
<redacted IP address in the response>

Does this mean envoy forward-proxy is only to be used for traffic coming from sidecar? and not when exposed via a public IP (like how squid can be used)?

[shibin-george]

Hi @alyssawilk , would you know what's going wrong here? Thanks!

[jewertow]

@shibin-george I tested your dynamic forward proxy on a kind cluster with metallb and it works fine. I think you need to configure an access log to find more information. If it won't be helpful, use wireshark to see what happens under the hood.
Do you run squid on the same k8s cluster?

Does this mean envoy forward-proxy is only to be used for traffic coming from sidecar?

No, it's a general purpose proxy.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

[dannyxu2015]

I'm not sure what envoy you are running but with the same config you shared I can do the test.

curl -x 127.0.0.1:10000 https://api.ipify.org -v
*   Trying 127.0.0.1:10000...
* Connected to 127.0.0.1 (127.0.0.1) port 10000 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to api.ipify.org:443
> CONNECT api.ipify.org:443 HTTP/1.1
> Host: api.ipify.org:443
> User-Agent: curl/7.79.1
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK

My envoy version

envoy --version

envoy  version: edd69583372955fdfa0b8ca3820dd7312c094e46/1.23.1/Modified/RELEASE/BoringSSL

I've tried your config, but got 503 error:

curl -x http://localhost:10000 https://google.com -v                                                                                                                                               56 ↵
*   Trying 127.0.0.1:10000...
* Connected to localhost (127.0.0.1) port 10000 (#0)
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Establish HTTP proxy tunnel to google.com:443
> CONNECT google.com:443 HTTP/1.1
> Host: google.com:443
> User-Agent: curl/8.1.2
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 503 Service Unavailable
< content-length: 22
< content-type: text/plain
< date: Fri, 08 Sep 2023 10:44:20 GMT
< server: envoy
< connection: close
<
* CONNECT tunnel failed, response 503
* Closing connection 0
curl: (56) CONNECT tunnel failed, response 503

my envoy version is

envoy  version: c7e8e7356d3a969c1b8e4e1f2687699acd91c6a1/1.26.1/Distribution/RELEASE/BoringSSL

[dannyxu2015]

@batchamalick any comment?