From 88b853a8aaa8a7ded7ee5146e221d388ebbbdf4c Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Thu, 28 Mar 2024 11:47:02 -0400 Subject: [PATCH 1/3] docs: update service mesh integration docs for transparent proxy Update the service mesh integration docs to explain how Consul needs to be configured for transparent proxy. Update the walkthrough to assume that `transparent_proxy` mode is the best approach, and move the manually-configured `upstreams` to a separate section for users who don't want to use Consul DNS. Ref: https://github.com/hashicorp/nomad/pull/20175 Ref: https://github.com/hashicorp/nomad/pull/20241 --- .../docs/integrations/consul/service-mesh.mdx | 126 +++++++++++++++--- 1 file changed, 109 insertions(+), 17 deletions(-) diff --git a/website/content/docs/integrations/consul/service-mesh.mdx b/website/content/docs/integrations/consul/service-mesh.mdx index 707719585bcc..43898130f192 100644 --- a/website/content/docs/integrations/consul/service-mesh.mdx +++ b/website/content/docs/integrations/consul/service-mesh.mdx @@ -134,6 +134,25 @@ service_prefix "" { policy = "read" } node_prefix "" { policy = "read" } ``` +#### Consul DNS For Transparent Proxy + +To use Consul's [transparent proxy][] mode with virtual IP addresses, you'll +need to configure Consul's DNS listener to be exposed to the workload network +namespace. You can do this without exposing the Consul agent on a public IP by +setting the Consul `bind_addr` to bind on a private IP address. + +If you want workloads to be able to use both Consul DNS and query external DNS +entries, you'll need to add [`recursors`][] for the external nameservers. + +For example, a HCL configuration with a [go-sockaddr/template][] binding to the +subnet `10.37.105.0/20`, with recursive DNS set to OpenDNS nameservers: + +```hcl +bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.37.105.0/20\" | limit 1 | attr \"address\" }}" + +recursors = ["208.67.222.222", "208.67.220.220"] +``` + ### Nomad Nomad must schedule onto a routable interface in order for the proxies to @@ -150,10 +169,14 @@ Nomad uses CNI reference plugins to configure the network namespace used to secu Consul service mesh sidecar proxy. All Nomad client nodes using network namespaces must have these CNI plugins [installed][cni_install]. +To use [`transparent_proxy`][] mode, Nomad client nodes will also need the +[`consul-cni`][] plugin installed. + ## Run the Service Mesh-enabled Services -Once Nomad and Consul are running, submit the following service mesh-enabled services -to Nomad by copying the HCL into a file named `servicemesh.nomad.hcl` and running: +Once Nomad and Consul are running, with Consul DNS enabled for transparent proxy +mode as described above, submit the following service mesh-enabled services to +Nomad by copying the HCL into a file named `servicemesh.nomad.hcl` and running: `nomad job run servicemesh.nomad.hcl` ```hcl @@ -170,7 +193,11 @@ job "countdash" { port = "9001" connect { - sidecar_service {} + sidecar_service { + proxy { + transparent_proxy {} + } + } } } @@ -200,10 +227,7 @@ job "countdash" { connect { sidecar_service { proxy { - upstreams { - destination_name = "count-api" - local_bind_port = 8080 - } + transparent_proxy {} } } } @@ -213,7 +237,7 @@ job "countdash" { driver = "docker" env { - COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}" + COUNTING_SERVICE_URL = "http://count-api.virtual.consul:9001" } config { @@ -240,8 +264,10 @@ The API service is defined as a task group with a bridge network: } ``` -Since the API service is only accessible via Consul service mesh, it does not define -any ports in its network. The service block enables service mesh. +Since the API service is only accessible via Consul service mesh, it does not +define any ports in its network. The `connect` block enables the service mesh +and the `transparent_proxy` block ensures that the service will be reachable via +a virtual IP address when used with Consul DNS. ```hcl group "api" { @@ -253,7 +279,11 @@ any ports in its network. The service block enables service mesh. port = "9001" connect { - sidecar_service {} + sidecar_service { + proxy { + transparent_proxy {} + } + } } } @@ -299,6 +329,64 @@ This allows you to connect to the web frontend in a browser by visiting The web frontend connects to the API service via Consul service mesh. +```hcl + service { + name = "count-dashboard" + port = "http" + + connect { + sidecar_service { + proxy { + transparent_proxy {} + } + } + } + } +``` + +The `connect` block with `transparent_proxy` configures the web frontend's +network namespace to route all access to the `count-api` service through the +Envoy proxy. + +The web frontend is configured to communicate with the API service with an +environment variable `$COUNTING_SERVICE_URL`: + +```hcl + env { + COUNTING_SERVICE_URL = "http://count-api.virtual.consul:9001" + } +``` + +The `transparent_proxy` block ensures that DNS queries are made to Consul so +that the `count-api.virtual.consul` name resolves to a virtual IP address. + +### Manually Configured Upstreams + +If you don't want to use Consul DNS and `transparent_proxy` mode, you can add +`upstream` blocks to the job spec. In that case, you don't need the +`transparent_proxy` block for the `count-api` service: + +```hcl + group "api" { + + # ... + + service { + name = "count-api" + port = "9001" + + connect { + sidecar_service {} + } + } + + # ... + + } +``` + +But you'll need to add an `upstreams` block to the `count-dashboard` service: + ```hcl service { name = "count-dashboard" @@ -320,8 +408,8 @@ The web frontend connects to the API service via Consul service mesh. The `upstreams` block defines the remote service to access (`count-api`) and what port to expose that service on inside the network namespace (`8080`). -The web frontend is configured to communicate with the API service with an -environment variable: +The web frontend will also need to use an environment variable to communicate +with the API service: ```hcl env { @@ -329,10 +417,9 @@ environment variable: } ``` -The web frontend is configured via the `$COUNTING_SERVICE_URL`, so you must -interpolate the upstream's address into that environment variable. Note that -dashes (`-`) are converted to underscores (`_`) in environment variables so -`count-api` becomes `count_api`. +This environment variable value gets interpolated with the upstream's +address. Note that dashes (`-`) are converted to underscores (`_`) in +environment variables so `count-api` becomes `count_api`. ## Limitations @@ -377,3 +464,8 @@ filesystem. [consul_ports]: /consul/docs/agent/config/config-files#ports [consul_grpc_tls]: /consul/docs/upgrading/upgrade-specific#changes-to-grpc-tls-configuration [cni_install]: /nomad/docs/install#post-installation-steps +[transparent proxy]: /consul/docs/k8s/connect/transparent-proxy +[go-sockaddr/template]: https://godoc.org/github.com/hashicorp/go-sockaddr/template +[`recursors`]: /consul/docs/agent/config/config-files#recursors +[`transparent_proxy`]: /nomad/docs/job-specification/transparent_proxy +[`consul-cni`]: https://releases.hashicorp.com/consul-cni From e5faef344c4a678e2dbaee33812f4d653adcedd1 Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Thu, 28 Mar 2024 15:10:20 -0400 Subject: [PATCH 2/3] note we don't need upstream service port, either --- website/content/docs/integrations/consul/service-mesh.mdx | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/website/content/docs/integrations/consul/service-mesh.mdx b/website/content/docs/integrations/consul/service-mesh.mdx index 43898130f192..2b37184b4ca0 100644 --- a/website/content/docs/integrations/consul/service-mesh.mdx +++ b/website/content/docs/integrations/consul/service-mesh.mdx @@ -237,7 +237,7 @@ job "countdash" { driver = "docker" env { - COUNTING_SERVICE_URL = "http://count-api.virtual.consul:9001" + COUNTING_SERVICE_URL = "http://count-api.virtual.consul" } config { @@ -353,12 +353,14 @@ environment variable `$COUNTING_SERVICE_URL`: ```hcl env { - COUNTING_SERVICE_URL = "http://count-api.virtual.consul:9001" + COUNTING_SERVICE_URL = "http://count-api.virtual.consul" } ``` The `transparent_proxy` block ensures that DNS queries are made to Consul so -that the `count-api.virtual.consul` name resolves to a virtual IP address. +that the `count-api.virtual.consul` name resolves to a virtual IP address. Note +that you don't need to specify a port number because the virtual IP will only be +directed to the correct service port. ### Manually Configured Upstreams From ae5fcb116a67fecd0a9b49a869202c44a8cbeee7 Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Wed, 3 Apr 2024 16:14:11 -0400 Subject: [PATCH 3/3] move section over from the reference docs --- .../docs/integrations/consul/service-mesh.mdx | 41 +++++++++++++++---- website/content/docs/networking/cni.mdx | 5 +++ 2 files changed, 37 insertions(+), 9 deletions(-) diff --git a/website/content/docs/integrations/consul/service-mesh.mdx b/website/content/docs/integrations/consul/service-mesh.mdx index 2b37184b4ca0..3df95e6b24e8 100644 --- a/website/content/docs/integrations/consul/service-mesh.mdx +++ b/website/content/docs/integrations/consul/service-mesh.mdx @@ -134,15 +134,33 @@ service_prefix "" { policy = "read" } node_prefix "" { policy = "read" } ``` -#### Consul DNS For Transparent Proxy - -To use Consul's [transparent proxy][] mode with virtual IP addresses, you'll -need to configure Consul's DNS listener to be exposed to the workload network -namespace. You can do this without exposing the Consul agent on a public IP by -setting the Consul `bind_addr` to bind on a private IP address. - -If you want workloads to be able to use both Consul DNS and query external DNS -entries, you'll need to add [`recursors`][] for the external nameservers. +#### Transparent Proxy + +Using Nomad's support for [transparent proxy][] configures the network namespace +so that traffic flows through the Envoy proxy. When the [`transparent_proxy`][] +block is enabled: + +* Nomad will invoke the [`consul-cni`][] CNI plugin to configure `iptables` rules + in the network namespace to force outbound traffic from an allocation to flow + through the proxy. +* If the local Consul agent is serving DNS, Nomad will set the IP address of the + Consul agent as the nameserver in the task's `/etc/resolv.conf`. +* Consul will provide a [virtual IP][] for any upstream service the workload + has access to, based on the service intentions. + +Using transparent proxy has several important requirements: + +* You must have the [`consul-cni`][] CNI plugin installed on the client host + along with the usual [required CNI plugins][cni_plugins]. +* To use Consul DNS and virtual IPs, you'll need to configure Consul's DNS + listener to be exposed to the workload network namespace. You can do this + without exposing the Consul agent on a public IP by setting the Consul + `bind_addr` to bind on a private IP address (the default is to use the + `client_addr`). +* The Consul agent must be configured with [`recursors`][] if you want + allocations to make DNS queries for applications outside the service mesh. +* You cannot set a [`network.dns`][] block on the allocation (unless you set + [`no_dns`][tproxy_no_dns], see below). For example, a HCL configuration with a [go-sockaddr/template][] binding to the subnet `10.37.105.0/20`, with recursive DNS set to OpenDNS nameservers: @@ -470,4 +488,9 @@ filesystem. [go-sockaddr/template]: https://godoc.org/github.com/hashicorp/go-sockaddr/template [`recursors`]: /consul/docs/agent/config/config-files#recursors [`transparent_proxy`]: /nomad/docs/job-specification/transparent_proxy +[tproxy_no_dns]: /nomad/docs/job-specification/transparent_proxy#no_dns [`consul-cni`]: https://releases.hashicorp.com/consul-cni +[virtual IP]: /consul/docs/services/discovery/dns-static-lookups#service-virtual-ip-lookups +[cni_plugins]: /nomad/docs/networking/cni#cni-reference-plugins +[consul_dns_port]: /consul/docs/agent/config/config-files#dns_port +[`network.dns`]: /nomad/docs/job-specification/network#dns-parameters diff --git a/website/content/docs/networking/cni.mdx b/website/content/docs/networking/cni.mdx index 64c47ac5e20a..2d8d2233ccb8 100644 --- a/website/content/docs/networking/cni.mdx +++ b/website/content/docs/networking/cni.mdx @@ -31,6 +31,9 @@ with Consul service mesh. See the Linux [post-install steps][cni_install] for installing CNI reference plugins. +To use Nomad's [`transparent_proxy`][] feature, you will also need the +[`consul-cni`][] plugin. + ## CNI plugins Spec-compliant plugins should work with Nomad, however, it's possible a plugin @@ -225,3 +228,5 @@ a unique value for your configuration. [loopback]: https://github.com/containernetworking/plugins#main-interface-creating [nomad_install]: /nomad/tutorials/get-started/get-started-install#post-installation-steps [portmap]: https://www.cni.dev/plugins/current/meta/portmap/ +[`transparent_proxy`]: /nomad/docs/job-specification/transparent_proxy +[`consul-cni`]: https://releases.hashicorp.com/consul-cni