Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: update service mesh integration docs for transparent proxy #20251

Merged
merged 3 commits into from
Apr 3, 2024
Merged

docs: update service mesh integration docs for transparent proxy #20251

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
88b853a
docs: update service mesh integration docs for transparent proxy
tgross Mar 28, 2024
e5faef3
note we don't need upstream service port, either
tgross Mar 28, 2024
ae5fcb1
move section over from the reference docs
tgross Apr 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
151 changes: 134 additions & 17 deletions website/content/docs/integrations/consul/service-mesh.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,43 @@ service_prefix "" { policy = "read" }
node_prefix "" { policy = "read" }
```

#### Transparent Proxy

Using Nomad's support for [transparent proxy][] configures the network namespace
so that traffic flows through the Envoy proxy. When the [`transparent_proxy`][]
block is enabled:

* Nomad will invoke the [`consul-cni`][] CNI plugin to configure `iptables` rules
in the network namespace to force outbound traffic from an allocation to flow
through the proxy.
* If the local Consul agent is serving DNS, Nomad will set the IP address of the
Consul agent as the nameserver in the task's `/etc/resolv.conf`.
* Consul will provide a [virtual IP][] for any upstream service the workload
has access to, based on the service intentions.

Using transparent proxy has several important requirements:

* You must have the [`consul-cni`][] CNI plugin installed on the client host
along with the usual [required CNI plugins][cni_plugins].
* To use Consul DNS and virtual IPs, you'll need to configure Consul's DNS
listener to be exposed to the workload network namespace. You can do this
without exposing the Consul agent on a public IP by setting the Consul
`bind_addr` to bind on a private IP address (the default is to use the
`client_addr`).
* The Consul agent must be configured with [`recursors`][] if you want
allocations to make DNS queries for applications outside the service mesh.
* You cannot set a [`network.dns`][] block on the allocation (unless you set
[`no_dns`][tproxy_no_dns], see below).

For example, a HCL configuration with a [go-sockaddr/template][] binding to the
subnet `10.37.105.0/20`, with recursive DNS set to OpenDNS nameservers:

```hcl
bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.37.105.0/20\" | limit 1 | attr \"address\" }}"

recursors = ["208.67.222.222", "208.67.220.220"]
```

### Nomad

Nomad must schedule onto a routable interface in order for the proxies to
Expand All @@ -150,10 +187,14 @@ Nomad uses CNI reference plugins to configure the network namespace used to secu
Consul service mesh sidecar proxy. All Nomad client nodes using network namespaces
must have these CNI plugins [installed][cni_install].

To use [`transparent_proxy`][] mode, Nomad client nodes will also need the
[`consul-cni`][] plugin installed.

## Run the Service Mesh-enabled Services

Once Nomad and Consul are running, submit the following service mesh-enabled services
to Nomad by copying the HCL into a file named `servicemesh.nomad.hcl` and running:
Once Nomad and Consul are running, with Consul DNS enabled for transparent proxy
mode as described above, submit the following service mesh-enabled services to
Nomad by copying the HCL into a file named `servicemesh.nomad.hcl` and running:
`nomad job run servicemesh.nomad.hcl`

```hcl
Expand All @@ -170,7 +211,11 @@ job "countdash" {
port = "9001"

connect {
sidecar_service {}
sidecar_service {
proxy {
transparent_proxy {}
}
}
}
}

Expand Down Expand Up @@ -200,10 +245,7 @@ job "countdash" {
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "count-api"
local_bind_port = 8080
}
transparent_proxy {}
}
}
}
Expand All @@ -213,7 +255,7 @@ job "countdash" {
driver = "docker"

env {
COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
COUNTING_SERVICE_URL = "http://count-api.virtual.consul"
}

config {
Expand All @@ -240,8 +282,10 @@ The API service is defined as a task group with a bridge network:
}
```

Since the API service is only accessible via Consul service mesh, it does not define
any ports in its network. The service block enables service mesh.
Since the API service is only accessible via Consul service mesh, it does not
define any ports in its network. The `connect` block enables the service mesh
and the `transparent_proxy` block ensures that the service will be reachable via
a virtual IP address when used with Consul DNS.

```hcl
group "api" {
Expand All @@ -253,7 +297,11 @@ any ports in its network. The service block enables service mesh.
port = "9001"

connect {
sidecar_service {}
sidecar_service {
proxy {
transparent_proxy {}
}
}
}
}

Expand Down Expand Up @@ -299,6 +347,66 @@ This allows you to connect to the web frontend in a browser by visiting

The web frontend connects to the API service via Consul service mesh.

```hcl
service {
name = "count-dashboard"
port = "http"

connect {
sidecar_service {
proxy {
transparent_proxy {}
}
}
}
}
```

The `connect` block with `transparent_proxy` configures the web frontend's
network namespace to route all access to the `count-api` service through the
Envoy proxy.

The web frontend is configured to communicate with the API service with an
environment variable `$COUNTING_SERVICE_URL`:

```hcl
env {
COUNTING_SERVICE_URL = "http://count-api.virtual.consul"
}
```

The `transparent_proxy` block ensures that DNS queries are made to Consul so
that the `count-api.virtual.consul` name resolves to a virtual IP address. Note
that you don't need to specify a port number because the virtual IP will only be
directed to the correct service port.

### Manually Configured Upstreams

If you don't want to use Consul DNS and `transparent_proxy` mode, you can add
`upstream` blocks to the job spec. In that case, you don't need the
`transparent_proxy` block for the `count-api` service:

```hcl
group "api" {

# ...

service {
name = "count-api"
port = "9001"

connect {
sidecar_service {}
}
}

# ...

}
```

But you'll need to add an `upstreams` block to the `count-dashboard` service:

```hcl
service {
name = "count-dashboard"
Expand All @@ -320,19 +428,18 @@ The web frontend connects to the API service via Consul service mesh.
The `upstreams` block defines the remote service to access (`count-api`) and
what port to expose that service on inside the network namespace (`8080`).

The web frontend is configured to communicate with the API service with an
environment variable:
The web frontend will also need to use an environment variable to communicate
with the API service:

```hcl
env {
COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
}
```

The web frontend is configured via the `$COUNTING_SERVICE_URL`, so you must
interpolate the upstream's address into that environment variable. Note that
dashes (`-`) are converted to underscores (`_`) in environment variables so
`count-api` becomes `count_api`.
This environment variable value gets interpolated with the upstream's
address. Note that dashes (`-`) are converted to underscores (`_`) in
environment variables so `count-api` becomes `count_api`.

## Limitations

Expand Down Expand Up @@ -377,3 +484,13 @@ filesystem.
[consul_ports]: /consul/docs/agent/config/config-files#ports
[consul_grpc_tls]: /consul/docs/upgrading/upgrade-specific#changes-to-grpc-tls-configuration
[cni_install]: /nomad/docs/install#post-installation-steps
[transparent proxy]: /consul/docs/k8s/connect/transparent-proxy
[go-sockaddr/template]: https://godoc.org/github.com/hashicorp/go-sockaddr/template
[`recursors`]: /consul/docs/agent/config/config-files#recursors
[`transparent_proxy`]: /nomad/docs/job-specification/transparent_proxy
[tproxy_no_dns]: /nomad/docs/job-specification/transparent_proxy#no_dns
[`consul-cni`]: https://releases.hashicorp.com/consul-cni
[virtual IP]: /consul/docs/services/discovery/dns-static-lookups#service-virtual-ip-lookups
[cni_plugins]: /nomad/docs/networking/cni#cni-reference-plugins
[consul_dns_port]: /consul/docs/agent/config/config-files#dns_port
[`network.dns`]: /nomad/docs/job-specification/network#dns-parameters
5 changes: 5 additions & 0 deletions website/content/docs/networking/cni.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ with Consul service mesh.

See the Linux [post-install steps][cni_install] for installing CNI reference plugins.

To use Nomad's [`transparent_proxy`][] feature, you will also need the
[`consul-cni`][] plugin.

## CNI plugins

Spec-compliant plugins should work with Nomad, however, it's possible a plugin
Expand Down Expand Up @@ -225,3 +228,5 @@ a unique value for your configuration.
[loopback]: https://github.com/containernetworking/plugins#main-interface-creating
[nomad_install]: /nomad/tutorials/get-started/get-started-install#post-installation-steps
[portmap]: https://www.cni.dev/plugins/current/meta/portmap/
[`transparent_proxy`]: /nomad/docs/job-specification/transparent_proxy
[`consul-cni`]: https://releases.hashicorp.com/consul-cni
Loading