From 93cc2207f77023ffca488be0c2b1ab7f81a8f3c3 Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Wed, 3 Apr 2024 16:22:11 -0400 Subject: [PATCH] docs: update service mesh integration docs for transparent proxy (#20251) Update the service mesh integration docs to explain how Consul needs to be configured for transparent proxy. Update the walkthrough to assume that `transparent_proxy` mode is the best approach, and move the manually-configured `upstreams` to a separate section for users who don't want to use Consul DNS. Ref: https://github.com/hashicorp/nomad/pull/20175 Ref: https://github.com/hashicorp/nomad/pull/20241 --- .../docs/integrations/consul/service-mesh.mdx | 151 ++++++++++++++++-- website/content/docs/networking/cni.mdx | 5 + 2 files changed, 139 insertions(+), 17 deletions(-) diff --git a/website/content/docs/integrations/consul/service-mesh.mdx b/website/content/docs/integrations/consul/service-mesh.mdx index 707719585bcc..3df95e6b24e8 100644 --- a/website/content/docs/integrations/consul/service-mesh.mdx +++ b/website/content/docs/integrations/consul/service-mesh.mdx @@ -134,6 +134,43 @@ service_prefix "" { policy = "read" } node_prefix "" { policy = "read" } ``` +#### Transparent Proxy + +Using Nomad's support for [transparent proxy][] configures the network namespace +so that traffic flows through the Envoy proxy. When the [`transparent_proxy`][] +block is enabled: + +* Nomad will invoke the [`consul-cni`][] CNI plugin to configure `iptables` rules + in the network namespace to force outbound traffic from an allocation to flow + through the proxy. +* If the local Consul agent is serving DNS, Nomad will set the IP address of the + Consul agent as the nameserver in the task's `/etc/resolv.conf`. +* Consul will provide a [virtual IP][] for any upstream service the workload + has access to, based on the service intentions. + +Using transparent proxy has several important requirements: + +* You must have the [`consul-cni`][] CNI plugin installed on the client host + along with the usual [required CNI plugins][cni_plugins]. +* To use Consul DNS and virtual IPs, you'll need to configure Consul's DNS + listener to be exposed to the workload network namespace. You can do this + without exposing the Consul agent on a public IP by setting the Consul + `bind_addr` to bind on a private IP address (the default is to use the + `client_addr`). +* The Consul agent must be configured with [`recursors`][] if you want + allocations to make DNS queries for applications outside the service mesh. +* You cannot set a [`network.dns`][] block on the allocation (unless you set + [`no_dns`][tproxy_no_dns], see below). + +For example, a HCL configuration with a [go-sockaddr/template][] binding to the +subnet `10.37.105.0/20`, with recursive DNS set to OpenDNS nameservers: + +```hcl +bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.37.105.0/20\" | limit 1 | attr \"address\" }}" + +recursors = ["208.67.222.222", "208.67.220.220"] +``` + ### Nomad Nomad must schedule onto a routable interface in order for the proxies to @@ -150,10 +187,14 @@ Nomad uses CNI reference plugins to configure the network namespace used to secu Consul service mesh sidecar proxy. All Nomad client nodes using network namespaces must have these CNI plugins [installed][cni_install]. +To use [`transparent_proxy`][] mode, Nomad client nodes will also need the +[`consul-cni`][] plugin installed. + ## Run the Service Mesh-enabled Services -Once Nomad and Consul are running, submit the following service mesh-enabled services -to Nomad by copying the HCL into a file named `servicemesh.nomad.hcl` and running: +Once Nomad and Consul are running, with Consul DNS enabled for transparent proxy +mode as described above, submit the following service mesh-enabled services to +Nomad by copying the HCL into a file named `servicemesh.nomad.hcl` and running: `nomad job run servicemesh.nomad.hcl` ```hcl @@ -170,7 +211,11 @@ job "countdash" { port = "9001" connect { - sidecar_service {} + sidecar_service { + proxy { + transparent_proxy {} + } + } } } @@ -200,10 +245,7 @@ job "countdash" { connect { sidecar_service { proxy { - upstreams { - destination_name = "count-api" - local_bind_port = 8080 - } + transparent_proxy {} } } } @@ -213,7 +255,7 @@ job "countdash" { driver = "docker" env { - COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}" + COUNTING_SERVICE_URL = "http://count-api.virtual.consul" } config { @@ -240,8 +282,10 @@ The API service is defined as a task group with a bridge network: } ``` -Since the API service is only accessible via Consul service mesh, it does not define -any ports in its network. The service block enables service mesh. +Since the API service is only accessible via Consul service mesh, it does not +define any ports in its network. The `connect` block enables the service mesh +and the `transparent_proxy` block ensures that the service will be reachable via +a virtual IP address when used with Consul DNS. ```hcl group "api" { @@ -253,7 +297,11 @@ any ports in its network. The service block enables service mesh. port = "9001" connect { - sidecar_service {} + sidecar_service { + proxy { + transparent_proxy {} + } + } } } @@ -299,6 +347,66 @@ This allows you to connect to the web frontend in a browser by visiting The web frontend connects to the API service via Consul service mesh. +```hcl + service { + name = "count-dashboard" + port = "http" + + connect { + sidecar_service { + proxy { + transparent_proxy {} + } + } + } + } +``` + +The `connect` block with `transparent_proxy` configures the web frontend's +network namespace to route all access to the `count-api` service through the +Envoy proxy. + +The web frontend is configured to communicate with the API service with an +environment variable `$COUNTING_SERVICE_URL`: + +```hcl + env { + COUNTING_SERVICE_URL = "http://count-api.virtual.consul" + } +``` + +The `transparent_proxy` block ensures that DNS queries are made to Consul so +that the `count-api.virtual.consul` name resolves to a virtual IP address. Note +that you don't need to specify a port number because the virtual IP will only be +directed to the correct service port. + +### Manually Configured Upstreams + +If you don't want to use Consul DNS and `transparent_proxy` mode, you can add +`upstream` blocks to the job spec. In that case, you don't need the +`transparent_proxy` block for the `count-api` service: + +```hcl + group "api" { + + # ... + + service { + name = "count-api" + port = "9001" + + connect { + sidecar_service {} + } + } + + # ... + + } +``` + +But you'll need to add an `upstreams` block to the `count-dashboard` service: + ```hcl service { name = "count-dashboard" @@ -320,8 +428,8 @@ The web frontend connects to the API service via Consul service mesh. The `upstreams` block defines the remote service to access (`count-api`) and what port to expose that service on inside the network namespace (`8080`). -The web frontend is configured to communicate with the API service with an -environment variable: +The web frontend will also need to use an environment variable to communicate +with the API service: ```hcl env { @@ -329,10 +437,9 @@ environment variable: } ``` -The web frontend is configured via the `$COUNTING_SERVICE_URL`, so you must -interpolate the upstream's address into that environment variable. Note that -dashes (`-`) are converted to underscores (`_`) in environment variables so -`count-api` becomes `count_api`. +This environment variable value gets interpolated with the upstream's +address. Note that dashes (`-`) are converted to underscores (`_`) in +environment variables so `count-api` becomes `count_api`. ## Limitations @@ -377,3 +484,13 @@ filesystem. [consul_ports]: /consul/docs/agent/config/config-files#ports [consul_grpc_tls]: /consul/docs/upgrading/upgrade-specific#changes-to-grpc-tls-configuration [cni_install]: /nomad/docs/install#post-installation-steps +[transparent proxy]: /consul/docs/k8s/connect/transparent-proxy +[go-sockaddr/template]: https://godoc.org/github.com/hashicorp/go-sockaddr/template +[`recursors`]: /consul/docs/agent/config/config-files#recursors +[`transparent_proxy`]: /nomad/docs/job-specification/transparent_proxy +[tproxy_no_dns]: /nomad/docs/job-specification/transparent_proxy#no_dns +[`consul-cni`]: https://releases.hashicorp.com/consul-cni +[virtual IP]: /consul/docs/services/discovery/dns-static-lookups#service-virtual-ip-lookups +[cni_plugins]: /nomad/docs/networking/cni#cni-reference-plugins +[consul_dns_port]: /consul/docs/agent/config/config-files#dns_port +[`network.dns`]: /nomad/docs/job-specification/network#dns-parameters diff --git a/website/content/docs/networking/cni.mdx b/website/content/docs/networking/cni.mdx index 64c47ac5e20a..2d8d2233ccb8 100644 --- a/website/content/docs/networking/cni.mdx +++ b/website/content/docs/networking/cni.mdx @@ -31,6 +31,9 @@ with Consul service mesh. See the Linux [post-install steps][cni_install] for installing CNI reference plugins. +To use Nomad's [`transparent_proxy`][] feature, you will also need the +[`consul-cni`][] plugin. + ## CNI plugins Spec-compliant plugins should work with Nomad, however, it's possible a plugin @@ -225,3 +228,5 @@ a unique value for your configuration. [loopback]: https://github.com/containernetworking/plugins#main-interface-creating [nomad_install]: /nomad/tutorials/get-started/get-started-install#post-installation-steps [portmap]: https://www.cni.dev/plugins/current/meta/portmap/ +[`transparent_proxy`]: /nomad/docs/job-specification/transparent_proxy +[`consul-cni`]: https://releases.hashicorp.com/consul-cni