posts/kube-grafana-prometheus/

[giscus[bot]]

posts/kube-grafana-prometheus/

Documentation for Techno Tim YouTube videos for all your copy pasta needs!

https://technotim.live/posts/kube-grafana-prometheus/

[cranky-coder]

got this somewhat working. a bunch of the default graphs are empty. but i was curious. how do you get prometheus to scrape other workloads? I have a pod with a service configure and have the annotation

annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '3500'
prometheus.io/path: '/metrics'

but it's still not getting picked up and scraped. Any suggestions?

[TechBrain64]

Tim,
Love your channel and contributions. I've built a K3S cluster mostly after following much of your videos and docs. That said, I've deployed prometheus-stack using much of your instructions. However, your grafana ingressroute is only exposing the dashboard on port 80 even though I've implemented cert-manager and uncommented tls in manifest. I've tried a number of things all with no success. Can you offer any assistance?

[Regis-Loyaute]

Here is my ingress if that can help you:

---

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: grafana-ingress
namespace: monitoring
annotations:
kubernetes.io/ingress.class: traefik-external
spec:
entryPoints:
- websecure
routes:
- match: Host(grafana2.myhomelab.fr)
kind: Rule
services:
- name: grafana
port: 80
tls:
secretName: myhomelab-fr-productions-tls

[cannfoddr]

what to do with the extra_server_args - I am running a server built on your ansible playbook?

[Regis-Loyaute]

he said it in the video

[cannfoddr]

He said to redeploy or update a config file but I can't seem to find which ones and what to update

[CanisHelix]

I believe you need to do the following:

Edit /etc/systemd/system/k3s.service on your master nodes
nano /etc/systemd/system/k3s.service

Do each master one at a time to avoid downtime.

Append --kube-controller-manager-arg bind-address=0.0.0.0 --kube-proxy-arg metrics-bind-address=0.0.0.0 --kube-scheduler-arg bind-address=0.0.0.0 --etcd-expose-metrics true --kubelet-arg containerd=/run/k3s/containerd/containerd.sock to the command line used.

Reload the service
systemctl daemon-reload and systemctl restart k3s.service

[CanisHelix]

Rancher's UI installs the version from: https://github.com/rancher/charts/tree/release-v2.7.3/charts/rancher-monitoring/102.0.0%2Bup40.1.2 which states This chart is based on the upstream kube-prometheus-stack chart.

Given that there are potentially Rancher specific differences, which set of charts do you currently recommend to use?

[djonko]

I am facing this :

helm install -n monitoring prometheus prometheus-community/kube-prometheus-stack -f values.yaml
Error: INSTALLATION FAILED: create: failed to create: Secret "sh.helm.release.v1.prometheus.v1" is invalid: data: Too long: must have at most 1048576 bytes

[Regis-Loyaute]

How did you setup your secret ?

[djonko]

I used this guide https://technotim.live/posts/kube-grafana-prometheus/
I switch back to version 47.1.0 and it works . thanks

helm install -n monitoring prometheus prometheus-community/kube-prometheus-stack --version 47.1.0 -f values.yaml

[Robert-aug13]

sudo helm install -n monitoring prometheus prometheus-community/kube-prometheus-stack --version 47.1.0 -f /home/ubuntu/values.yml
Error: INSTALLATION FAILED: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp 127.0.0.1:8080: connect: connection refused

working on microk8s.

[Robert-aug13]

I was able to find out the issue. I had to use sudo microk8s helm..Cant use sudo helm install on microk8s.

[aamer59]

Hi Tim,

it's a perfect solution for monitoring and only thing lacking is persisting the data... it would be great if you could add data persistance piece of code. I did try to append it from the various sources but no luck. one or other error is poping up... Please spare few mins to add the pvc portion.

Thanks in advance.

Regards,
Ameer

[M0ses-T]

I still cannot find the values.yaml file

[pirateszero92]

now u can find the link ? it's under section "Create a values file to hold our helm values"
---->>>> paste in values from here

[pirateszero92]

a million thank for your solution, i have expose serivec to loadbalnacer it's perfect thank.

[SmokersCough]

i set up the K3s cluster using your ansible playbook, i was able to follow the sets to install prometheus and grafana got it up and running. thank you. But i can not get any information on the dashboards for the servers using endpoints in values file.
grafana showing nothing for ectd, scheduler, control manager and so on. should i use the vip address that was set up when i ran the k3s playbook?