From 48e93a1c46fe7190bc94d526c189e48020cc8720 Mon Sep 17 00:00:00 2001 From: Tim Gross Date: Fri, 16 Feb 2024 15:52:12 -0500 Subject: [PATCH] tproxy: `transparent_proxy` docs Ref: https://github.com/hashicorp/nomad/pull/20175 --- .../content/docs/job-specification/proxy.mdx | 57 ++++-- .../job-specification/transparent_proxy.mdx | 172 ++++++++++++++++++ .../docs/job-specification/upstreams.mdx | 18 +- 3 files changed, 225 insertions(+), 22 deletions(-) create mode 100644 website/content/docs/job-specification/transparent_proxy.mdx diff --git a/website/content/docs/job-specification/proxy.mdx b/website/content/docs/job-specification/proxy.mdx index 9b86faf3c26b..5aa84a511729 100644 --- a/website/content/docs/job-specification/proxy.mdx +++ b/website/content/docs/job-specification/proxy.mdx @@ -13,9 +13,8 @@ description: |- /> The `proxy` block allows configuring various options for the sidecar proxy -managed by Nomad for [Consul -Connect](/nomad/docs/integrations/consul-connect). It is valid only -within the context of a `sidecar_service` block. +managed by Nomad for [Consul Connect][]. It is valid only within the context of +a `sidecar_service` block. ```hcl job "countdash" { @@ -50,23 +49,29 @@ job "countdash" { ## `proxy` Parameters -- `local_service_address` `(string: "127.0.0.1")` - The address the local service binds to. Useful to - customize in clusters with mixed Connect and non-Connect services. +- `config` `(map: nil)` - Proxy configuration that is opaque to Nomad and passed + directly to Consul. See [Consul Connect documentation][envoy_dynamic_config] + for details. Keys and values support [runtime variable interpolation][]. +- `expose` ([expose]: nil) - Used to configure expose path + configuration for Envoy. See Consul's [Expose Paths Configuration + Reference][expose_path_ref] for more information. +- `local_service_address` `(string: "127.0.0.1")` - The address the local + service binds to. Useful to customize in clusters with mixed Connect and + non-Connect services. - `local_service_port` `(int: )` - The port the local service binds to. - Usually the same as the parent service's port, it is useful to customize in clusters with mixed - Connect and non-Connect services. -- `upstreams` ([upstreams][]: nil) - Used to configure details of each upstream service that - this sidecar proxy communicates with. -- `expose` ([expose]: nil) - Used to configure expose path configuration for Envoy. - See Consul's [Expose Paths Configuration Reference](/consul/docs/connect/registration/service-registration#expose-paths-configuration-reference) - for more information. -- `config` `(map: nil)` - Proxy configuration that is opaque to Nomad and - passed directly to Consul. See [Consul Connect documentation](/consul/docs/connect/proxies/envoy#dynamic-configuration) - for details. Keys and values support [runtime variable interpolation][interpolation]. + Usually the same as the parent service's port, it is useful to customize in + clusters with mixed Connect and non-Connect services. +- `transparent_proxy` ([transparent_proxy][]: nil) - Used to enable + [transparent proxy][tproxy] mode, which allows the proxy to use Consul service + intentions to automatically configure upstreams, and configures iptables rules + to force traffic from the allocation to flow through the proxy. +- `upstreams` ([upstreams][]: nil) - Used to configure details of + each upstream service that this sidecar proxy communicates with. ## `proxy` Examples -The following example is a proxy specification that includes upstreams configuration. +The following example is a proxy specification that includes upstreams +configuration. ```hcl sidecar_service { @@ -79,10 +84,28 @@ sidecar_service { } ``` +The following example is a proxy specification that includes transparent proxy +configuration. Note that with transparent proxy, you will not need to configure +an `upstreams` block. + +```hcl +sidecar_service { + proxy { + transparent_proxy { + } + } +} +``` + +[Consul Connect]: /nomad/docs/integrations/consul-connect [job]: /nomad/docs/job-specification/job 'Nomad job Job Specification' [group]: /nomad/docs/job-specification/group 'Nomad group Job Specification' [task]: /nomad/docs/job-specification/task 'Nomad task Job Specification' -[interpolation]: /nomad/docs/runtime/interpolation 'Nomad interpolation' +[runtime variable interpolation]: /nomad/docs/runtime/interpolation 'Nomad interpolation' [sidecar_service]: /nomad/docs/job-specification/sidecar_service 'Nomad sidecar service Specification' [upstreams]: /nomad/docs/job-specification/upstreams 'Nomad upstream config Specification' [expose]: /nomad/docs/job-specification/expose 'Nomad proxy expose configuration' +[envoy_dynamic_config]: /consul/docs/connect/proxies/envoy#dynamic-configuration +[expose_path_ref]: /consul/docs/connect/registration/service-registration#expose-paths-configuration-reference +[transparent_proxy]: /nomad/docs/job-specification/transparent_proxy +[tproxy]: /consul/docs/k8s/connect/transparent-proxy diff --git a/website/content/docs/job-specification/transparent_proxy.mdx b/website/content/docs/job-specification/transparent_proxy.mdx new file mode 100644 index 000000000000..4e69997cffa0 --- /dev/null +++ b/website/content/docs/job-specification/transparent_proxy.mdx @@ -0,0 +1,172 @@ +--- +layout: docs +page_title: transparent_proxy Block - Job Specification +description: |- + The "transparent_proxy" block allows specifying options for configuring Envoy + in Consul Connect transparent proxy mode. +--- + +# `transparent_proxy` Block + + + +The `transparent_proxy` block configures the Envoy sidecar proxy to act as a +Consul Connect [transparent proxy][tproxy]. This simplifies the configuration of +Consul Connect by eliminating the need to configure [`upstreams`][] blocks in +Nomad. Instead, the Envoy proxy will takes its configuration entirely from +Consul [service intentions][]. When transparent proxy is enabled: + +* Nomad will invoke the [`consul-cni`][] CNI plugin to configure `iptables` rules + in the network namespace to force outbound traffic from an allocation to flow + through the proxy. +* If the local Consul agent is serving DNS, Nomad set the IP address of the + Consul agent as the nameserver in the task's `/etc/resolv.conf`. +* Consul will provide a [virtual IP][] for any upstream service the workload + has access to, based on the service intentions. + +Using transparent proxy has several important requirements: + +* You must have the [`consul-cni`][] CNI plugin installed on the client host + along with the usual [required CNI plugins][cni_plugins]. +* You can only have a single `connect` block in any task group that uses + transparent proxy. +* To use Consul DNS and virtual IPs, you must set the Consul agent's + [`ports.dns`][consul_dns_port] configuration to an IP other than localhost + (the default is to use the `client_addr`). +* The Consul agent must be configured with [`recursors`][] if you want + allocations to make DNS queries for applications outside the service mesh. + +## `transparent_proxy` Parameters + +* `exclude_inbound_ports` `([]string: nil)` - A list of inbound ports to exclude + from the IP tables redirect rule. These ports can be specified as either + [network port labels][port_labels] or as numeric ports. Nomad will + automatically add the following to this list: + * The [`local_path_port`][] of any [`expose`][] block. + * The port of any service check with [`expose=true`][check_expose] set. + * The port of any `network.port` with a [`static`][] value. +* `exclude_outbound_cidrs` `([]string: nil)` - A list of CIDR subnets that + should be excluded from outbound traffic redirection. This allows traffic to + these subnets to bypass the Envoy proxy. +* `exclude_outbound_ports` `([]int: nil)` - A list of port numbers that should + be excluded from outbound traffic redirection. This allows traffic to these + subnets to bypass the Envoy proxy. +* `exclude_uids` `([]string: nil)` - A list of Unix user IDs (UIDs) that should + be excluded from outbound traffic redirection. When unset, only the Envoy + proxy's user will be allowed to bypass the iptables rule. +* `no_dns` `(bool: false)` - When set to true, do not set Consul as the + nameserver for the workload and do not create IP tables rules to allow DNS to + bypass the Envoy proxy. If you want DNS to work inside the allocation with + `no_dns=true`, you will need to add your own CIDR and port exclusions for your + DNS nameserver. +* `outbound_port` `(int: 15001)` - The port that Envoy will bind on inside the + network namespace. The iptables rules created by `consul-cni` will force + traffic to flow to this port. You should only set this value if you have + specifically set the [`outbound_listener_port`][] in your Consul proxy + configuration. You can change the default value for a given node via [client + metadata][#client-metadata] (see below). +* `uid` `(string "101")` - The Unix user ID (UID) used by the Envoy proxy. You + should only set this value if you have a custom build of the Envoy container + image which uses a different UID. You can change the default value for a given + node via [client metadata][#client-metadata] (see below). + +## Client Metadata + +You can change the default [`outbound_port`][#outbound_port] and [`uid`][#uid] +for a given client node by updating the node metadata via the [`nomad node meta +apply`][] command. The attributes that can be updated are: + +* `connect.transparent_proxy.default_uid`: Sets the default value of + [`uid`][#uid] for this node. +* `connect.transparent_proxy.default_outbound_port`: Sets the default value of + [`outbound_port`][#outbound_port] for this node. + +For example, to set the default value for the `uid` field to 120: + +```shell-session +$ nomad node meta apply connect.transparent_proxy.default_uid=120 +$ nomad node meta read -json | jq -r '.Dynamic | ."connect.transparent_proxy.default_uid"' +120 +``` + +You should not normally need to set these values unless you are using custom +Envoy images. + +## Examples + +### Minimal Example + +The following example is a minimal transparent proxy specification. Note that +with transparent proxy, you will not need to configure an `upstreams` block. + +```hcl +sidecar_service { + proxy { + transparent_proxy { + } + } +} +``` + +If you had a downstream task group `count-dashboard` that needed to connect to +an upstream task group `count-api` listening on port 9001, you could create a +Consul service intention with the following specification: + +```hcl +Kind = "service-intentions" +Name = "count-api" +Sources = [ + { + Name = "count-dashboard" + Action = "allow" + } +] +``` + +And then the downstream service `count-dashboard` could reach the `count-api` +service by making requests to `http://count-api.virtual.consul:9001`. + +### External DNS + +The following example is a transparent proxy specification where external DNS is +used. To find the address of other allocations in this configuration, you will +need to use a [`template`][] block to query Consul. + +```hcl +sidecar_service { + proxy { + transparent_proxy { + excluded_outbound_ports = [53] + excluded_outbound_cidrs = ["208.67.222.222/32", "208.67.220.220/32"] + no_dns = true + } + } +} +``` + +[tproxy]: /consul/docs/k8s/connect/transparent-proxy +[`upstreams`]: /nomad/docs/job-specification/upstreams +[service intentions]: /consul/docs/connect/config-entries/service-intentions +[virtual IP]: /consul/docs/services/discovery/dns-static-lookups#service-virtual-ip-lookups +[`consul-cni`]: https://releases.hashicorp.com/consul-cni +[cni_plugins]: /nomad/docs/networking/cni#cni-reference-plugins +[consul_dns_port]: /consul/docs/agent/config/config-files#dns_port +[`recursors`]: /consul/docs/agent/config/config-files#recursors +[port_labels]: /nomad/docs/job-specification/network#port-parameters +[`local_path_port`]: /nomad/docs/job-specification/expose#local_path_port +[`expose`]: /nomad/docs/job-specification/expose +[check_expose]: /nomad/docs/job-specification/service#expose +[`static`]: /nomad/docs/job-specification/network#static +[`outbound_listener_port`]: /consul/docs/connect/proxies/proxy-config-reference#outbound_listener_port +[`template`]: /nomad/docs/job-specification/template#consul-integration +[`nomad node meta apply`]: /nomad/docs/commands/node/meta/apply diff --git a/website/content/docs/job-specification/upstreams.mdx b/website/content/docs/job-specification/upstreams.mdx index 102cefb6d9bb..07297aae8b70 100644 --- a/website/content/docs/job-specification/upstreams.mdx +++ b/website/content/docs/job-specification/upstreams.mdx @@ -21,12 +21,12 @@ description: |- /> The `upstreams` block allows configuring various options for managing upstream -services that a [Consul -Connect](/nomad/docs/integrations/consul-connect) proxy routes to. It -is valid only within the context of a `proxy` block. +services that a [Consul Connect][] proxy routes to. It is valid only within the +context of a `proxy` block. -For Consul-specific details see the [Consul Connect -Guide](/consul/tutorials/get-started-vms/virtual-machine-gs-service-discovery). +For Consul-specific details see the [Consul Connect Guide][]. Note that using +`upstream` may not be necessary if you have configured the proxy with the +[`transparent_proxy`][] block. ```hcl job "countdash" { @@ -135,6 +135,14 @@ and a local bind port. } ``` +The `upstreams` block allows configuring various options for managing upstream +services that a [] proxy routes to. It is valid only within the +context of a `proxy` block. + + +[Consul Connect]: /nomad/docs/integrations/consul-connect +[Consul Connect Guide]: /consul/tutorials/get-started-vms/virtual-machine-gs-service-discovery +[`transparent_proxy`]: /nomad/docs/job-specification/transparent_proxy [job]: /nomad/docs/job-specification/job 'Nomad job Job Specification' [group]: /nomad/docs/job-specification/group 'Nomad group Job Specification' [task]: /nomad/docs/job-specification/task 'Nomad task Job Specification'