prometheus-operator on Minikube / Accessing services

[hackermann]

First of all, I am a complete Newbie in K8s / Prometheus, and it seems I am doing something terribly wrong. I did the setup of the prometheus-operator as described in 'Quickstart', now I can access the dashboards, but I cannot access my own services anymore. I applied this minimal example:

apiVersion: v1
kind: Pod
metadata:
  name: nginxserver
  labels:
    app: nginxserver
spec:
  containers:
    - name: testpod0
      image: nginx
      ports:
      - containerPort: 80
        protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: nginxserverservice
spec:
  selector:
    app: nginxserver
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: NodePort

Then:

minikube service nginxserverservice --url

The service is not available. Then I type:

kubectl delete -f manifests/

Now I can access the service again. I'm pretty sure this is not a bug (right?), because it is such a minimal example. But what am I doing wrong? (Again, I'm a complete newbie, just wanted to play around a bit with Prometheus, I already read the complete README.md, but nothing useful there ..)

[ArthurSens]

Hey, it looks like you only missed the nodePort field on your port 🙂
https://kubernetes.io/docs/concepts/services-networking/service/#nodeport.

E.g.:

apiVersion: v1
kind: Service
metadata:
  name: nginxserverservice
spec:
  selector:
    app: nginxserver
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30007
  type: NodePort

Or you could just change the service type:

apiVersion: v1
kind: Service
metadata:
  name: nginxserverservice
spec:
  selector:
    app: nginxserver
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

You can always find out why services didn't start with the command: kubectl describe service <service-name>

[hackermann]

Hey, thanks for your answer, but unfortunately it is still not working. The service is actually starting, I just cannot access it (e.g. in a browser).

This is the output of describe service when I use type ClusterIP (no difference, if kube-prometheus is running or not):

Name:              nginxserverservice
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=nginxserver
Type:              ClusterIP
IP:                10.106.135.3
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         172.17.0.12:80
Session Affinity:  None
Events:            <none>

If I try to wget the service from another pod, I get this output as example:

wget -v 10.106.135.3:80
--2021-03-31 07:52:38--  http://10.106.135.3/
Connecting to 10.106.135.3:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 612 [text/html]
Saving to: 'index.html'

index.html                    100%[=================================================>]     612  --.-KB/s    in 0s

2021-03-31 07:52:38 (130 MB/s) - 'index.html' saved [612/612]

Then I start the kube-prometheus, delete and start the service again and wget gives me (with the new assigned service-IP):

wget -v 10.106.235.208:80
--2021-03-31 07:55:28--  http://10.106.235.208/
Connecting to 10.106.235.208:80...

But you think, this should work, right? Then maybe this is a bug??

What I tried so far is:

• kube-prometheus Version 0.7.0 and HEAD
• deleting and re-starting minikube multiple times
• minikube with --kubernetes-version=v1.20.0 (as described in Readme.MD) and the default version (which was v1.20.2 if I remember correctly)

[hackermann]

Now it is getting strange. So far I was running minikube on Windows 10. I tried the exact same configuration with a minikube on debian -- and it works. The services are still accessible after starting kube-prometheus. Both were running on --kubernetes-version=v1.20.0 and with docker as engine. Any ideas what the problem might be, or how to debug it further?