VHOST HTTP 503 Troubleshooting

The first step to do when troubleshooting Marathon-LB is to look at the active haproxy configuration. This can be obtained by looking at your public agent IP address, URL http://<public-agent-ip>:9090/_haproxy_getconfig

The key portion is the set of frontends and backends:

frontend marathon_http_in
  bind *:80
  mode http
  use_backend %[req.hdr(host),lower,regsub(:.*$,,),map(/marathon-lb/domain2backend.map)]

frontend marathon_http_appid_in
  bind *:9091
  mode http
  use_backend %[req.hdr(x-marathon-app-id),lower,map(/marathon-lb/app2backend.map)]

frontend marathon_https_in
  bind *:443 ssl crt /etc/ssl/cert.pem
  mode http
  use_backend %[ssl_fc_sni,lower,map(/marathon-lb/domain2backend.map)]

frontend nginx_10000
  bind *:10000
  mode http
  use_backend nginx_10000

backend nginx_10000
  balance roundrobin
  mode http
  option forwardfor
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request add-header X-Forwarded-Proto https if { ssl_fc }
  server 10_10_0_92_6599 10.10.0.92:6599

When using Marathon-LB with a HAPROXY_{n}_VHOST field, then the rough operation of Marathon-LB is as follows:

• Marathon-LB creates a file called /marathon-lb/domain2backend.map inside the container, containing a mapping of hostnames to backends. You can see this by navigating to your public agent IP address, with URL http://<public-agent-ip>:9090/_haproxy_getvhostmap. It will look something like this:

test.local nginx_10000
<additional entries>

• When Marathon-LB receives an HTTP request on either port 80 or 443, it will look at the request headers to determine which backend to send it to:
  • If it comes in on port 80, it will use this line from the config: use_backend %[req.hdr(host),lower,regsub(:.*$,,),map(/marathon-lb/domain2backend.map)], which tells it to do the following:
    1. Look at the hostname in the HTTP header
    2. Convert it to lowercase.
    3. Remove the port with a regex substition (:.*$ replaced with ``)
    4. Look in the /marathon-lb/domain2backend.map file for a corresponding backend
    5. Use the identified backend.
  • If it comes in on port 443, it will use this line from the config: use_backend %[ssl_fc_sni,lower,map(/marathon-lb/domain2backend.map)], which tells it to do the following:
    1. Look at the "Server Name Indication" (SNI) TLS extension field from the incoming TLS session
    2. Convert it to lowercase
    3. Look in the /marathon-lb/domain2backend.map file for a corresponding backend
    4. Use the identified backend.

In order for the port 443 vhost to work, the SNI field must be populated and must match the desired vhost/hostname. Certain (older) clients do not support SNI, and if you use curl with the -k (the 'insecure' flag), certain versions of curl will not use it.

Here's an example and some test outputs If you have an app with this set of HAPROXY labels:

{
  "id": "/pep/pep-prod-secure-v1",
  ...
  "labels": {
    ...
    "HAPROXY_GROUP": "external",
    "HAPROXY_0_VHOST": "marathon-lb.marathon.autoip.dcos.thisdcos.directory, dkerrigan-publicsl-1874g5qdqkdqf-459830456.us-west-2.elb.amazonaws.com, somehostthatdoesnotexist.com"
  }
}

marathon-lb.marathon.autoip.dcos.thisdcos.directory Should work from inside of the cluster dkerrigan-publicsl-1874g5qdqkdqf-459830456.us-west-2.elb.amazonaws.com Is the public dns for my ELB, and should be accessible from outside the cluster somehostthatdoesnotexist.com Is not a valid vhost, I added it to test an inaccessible host using CURL

10.0.7.15 Is the internal IP for my marathon-lb instance

From inside the cluster, I performed the following commands:

## SUCCESSFUL: 200 OK

curl -k https://dkerrigan-publicsl-1874g5qdqkdqf-459830456.us-west-2.elb.amazonaws.com/
## SUCCESSFUL: 200 OK

curl -k https://10.0.7.15
## FAILURE: 503 (Expected because there is no vhost specified for that IP)

curl -k -H 'Host: marathon-lb.marathon.autoip.dcos.thisdcos.directory' https://10.0.7.15
## FAILURE: 503 (Expected because HTTPS Vhost checks are performed using SNI (server name indicator, not the Host header)

curl -k --resolve marathon-lb.marathon.autoip.dcos.thisdcos.directory:443:10.0.7.15 https://marathon-lb.marathon.autoip.dcos.thisdcos.directory
## SUCCESS: 200 OK (This was successful, but it is misleading because marathon-lb.marathon.autoip.dcos.thisdcos.directory is actually already resolvable from where I am running curl)

curl -k --resolve somehostthatdoesnotexist.com:443:10.0.7.15 https://somehostthatdoesnotexist.com
## SUCCESS: 200 OK

Summary: In order to trick curl into sending the correct SNI for the vhost to marathon-lb, please try this command from inside the cluster:

curl -k --resolve <vhost>:443:<marathon-lb internal ip> https://<vhost>/