Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add documentation for using TLS on the metric endpoint #1376

Merged
merged 4 commits into from
Jan 10, 2024
Merged

feat: add documentation for using TLS on the metric endpoint #1376

Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f46317d
feat: add tls config to metrics docs
ThatsMrTalbot Jan 4, 2024
b1e8c99
docs: add podmonitor example for tls metrics endpoint
ThatsMrTalbot Jan 9, 2024
f72d3aa
docs: add note that tls metrics certs must be mounted into the pod
ThatsMrTalbot Jan 9, 2024
74017d6
docs: update prometheus example to correct port name
ThatsMrTalbot Jan 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions content/docs/devops-tips/prometheus-metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,83 @@ spec:
honorLabels: true
```

### TLS

TLS can be enabled on the metrics endpoint for end-to-end encryption. This is achieved either using pre-signed static certificates, or using the internal dynamic certificate signing.

#### Static certificates

Static certificates can be provided to the cert-manager controller to use when listening on the metric endpoint. If the certificate files are changed then cert-manager will reload the certificates for zero-downtime rotation.

Static certificates can be specified via the flags `--metrics-tls-cert-file` and `--metrics-tls-private-key-file` or the corresponding config file parameters `metricsTLSConfig.filesystem.certFile` and `metricsTLSConfig.filesystem.keyFile`.

The certificate and private key must be mounted into the controller pod for this to work, if cert-manager is deployed using helm the `.volumes[]` and `.mounts[]` properties can facilitate this.

An example config file would be:

```yaml
apiVersion: controller.config.cert-manager.io/v1alpha1
kind: ControllerConfiguration
metricsTLSConfig:
filesystem:
certFile: "/path/to/cert.pem"
keyFile: "/path/to/key.pem"
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't tried this yet, but how would you get these files into the Pod filesystem?
Using the .volumes and .mounts Helm values?

And if so, how would the user manage those Certifcates and keys?
It seems like the answer should be to use a cert-manager itself to create a Secret and mount that,
but there will be a bootstrapping problem.

So maybe we should explain that here in the docs.

Copy link
Contributor Author

@ThatsMrTalbot ThatsMrTalbot Jan 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can indeed pre-create them and use volumes and mounts to add to the filesystem.

I also worked out that if you can use cert-manager itself without having a bootstrap problem (kind of, the metrics endpoint will not work for a short while)

In Kubernetes you can mark a secret volume as optional, this means the pod is not blocked from starting if the secret is missing. Then cert manager can be used to provision the secret, kubelet would eventually notice the secret now exists and mount it in the correct place where cert manager would pick it up.

Not sure we want to recommend that way though

I can certainly add a comment saying you would need the certs mounted via a volume, with a "see .volumes and .mounts helm values for information on how to do this" line

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a note that the certificate and key must be mounted into the pod


#### Dynamic certificates

In this mode cert-manager will create a CA in a named secret, then use this CA to sign the metrics endpoint certificate. This mode will also take care of rotation, auto rotating the certificate as required.

Dynamic certificates can be specified via the flags `--metrics-dynamic-serving-ca-secret-namespace`, `--metrics-dynamic-serving-ca-secret-name` and `--metrics-dynamic-serving-dns-names` or the corresponding config file parameters `metricsTLSConfig.dynamic.secretNamespace`, `metricsTLSConfig.dynamic.secretName` and `metricsTLSConfig.dynamic.dnsNames`.

An example config file would be:

```yaml
apiVersion: controller.config.cert-manager.io/v1alpha1
kind: ControllerConfiguration
metricsTLSConfig:
dynamic:
secretNamespace: "cert-manager"
secretName: "cert-manager-metrics-ca"
dnsNames:
- cert-manager-metrics
- cert-manager-metrics.cert-manager
- cert-manager-metrics.cert-manager.svc
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This worked, but how should I configure my client with the CA certificate?
The problem is that the Secret resource contains the CA certificate and the private key so I don't really want to mount that Secret into my prometheus client Pod.

I could implement some other script that copies the CA certificate to another ConfigMap, but the CA is only valid for 1 year, so I'd need to schedule the script to run at the time the CA certificate gets renewed.

Also, I'm unsure about whether these DNS names are useful.
We recommend users to run two replicas of the controller, and
I suppose that my prometheus client needs to collect metrics from each of the replicas.
These names would resolve to the loadbalancer service IP which would round robin between the Pods.

Seems like the IP addresses of the Pods should also be added to the serving certificate too,
so that the prometheus client can connect to the Pod by IP address.
Is that how Prometheus will operate if using a PodMonitor resource (as recommended in the section above this)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are using prometheus operator, you can point reference a Secret as part of a PodMonitor or ServiceMonitor object https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.TLSConfig

The way Prometheus operator implements this is to copy the referenced secret keys into the Prometheus secret, since only the keys specified are copied, the private key would not be present inside the prometheus pod without a mis-configuration (see https://github.com/prometheus-operator/prometheus-operator/blob/547fea903e3df86272573e20c3b1479d0557a1a3/pkg/assets/store.go for implementation).

PodMonitors also let you set the serverName you wish to use when validating, which means you don't strictly need the IP address, but I can also see the benefit in including it in the cert

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added an example on how PodMonitor can be used to trust the CA and validate the serverName

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding that example and thanks for helping me get prometheus set up to test this.

I finally got it working and can see the cert-manager targets in the Prometheus Web UI,
using HTTPS URLs.

image

In that case I patched the ServiceMonitor resource which is installed by the Helm chart:

$ git diff
diff --git a/service-monitor.live.yaml b/service-monitor.live.yaml
index b77ffb52f..1aa6a3992 100644
--- a/service-monitor.live.yaml
+++ b/service-monitor.live.yaml
@@ -22,5 +22,12 @@ spec:
   - targetPort: 9402
     path: /metrics
     interval: 60s
+    scheme: https
     scrapeTimeout: 30s
     honorLabels: false
+    tlsConfig:
+      ca:
+        secret:
+          key: tls.crt
+          name: cert-manager-metrics-ca
+      serverName: cert-manager-metrics


When using Prometheus the CA generated by the generated certificate authority can be trusted as part of the `PodMonitor` or `ServiceMonitor` spec:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: cert-manager
namespace: cert-manager
labels:
app: cert-manager
app.kubernetes.io/name: cert-manager
app.kubernetes.io/instance: cert-manager
app.kubernetes.io/component: "controller"
spec:
jobLabel: app.kubernetes.io/name
selector:
matchLabels:
app: cert-manager
app.kubernetes.io/name: cert-manager
app.kubernetes.io/instance: cert-manager
app.kubernetes.io/component: "controller"
podMetricsEndpoints:
- port: http
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this port name should be "http-metrics", here and in the example above.

$ kubectl -n cert-manager get deploy cert-manager -o yaml | grep -A6 ports
        ports:
        - containerPort: 9402
          name: http-metrics
          protocol: TCP
        - containerPort: 9403
          name: http-healthz
          protocol: TCP

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, thats what the helm chart names the port. I've updated docs to reflect this.

honorLabels: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also added scheme: https because the API docs say that the default is http....but perhaps the default changes based on the existence of a tlsConfig stanza?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yea, added that to the docs as its probably required

# TLS config trusting the CA and specifying the server name
tlsConfig:
serverName: cert-manager-metrics
ca:
secret:
name: cert-manager-metrics-ca
key: "tls.crt"
```

## Monitoring Mixin

Monitoring mixins are a way to bundle common alerts, rules, and dashboards for an application in a configurable and extensible way, using the Jsonnet data templating language. A cert-manager monitoring mixin can be found here https://gitlab.com/uneeq-oss/cert-manager-mixin. Documentation on usage can be found with the `cert-manager-mixin` project.