Skip to content

Latest commit

 

History

History
133 lines (90 loc) · 7.43 KB

monitoring.md

File metadata and controls

133 lines (90 loc) · 7.43 KB
Raw

Monitoring Kubernetes Clusters

Monitoring your Kubernetes cluster lets you see its health and performance. Statistics such as CPU, memory, and disk usage are available for both Linux and Windows nodes in your AKS Engine cluster.

Resource metrics are collected by the lightweight, in-memory metrics-server component. Metrics-server discovers nodes and queries each one's kubelet for CPU and memory usage.

Metrics-server is part of every AKS Engine deployment. To verify that metrics-server is running in your cluster:

$ kubectl get pods --namespace=kube-system -l k8s-app=metrics-server
NAME                             READY   STATUS    RESTARTS   AGE
metrics-server-bb7db87bc-nm6vn   1/1     Running   2          140m

Tools like kubectl and the Kubernetes Dashboard use metrics-server, and it has an API to get metrics for your custom monitoring solution.

Monitoring Options

  1. kubectl
  2. Kubernetes Dashboard
  3. Azure Monitor for containers
  4. Monitoring extension

Kubectl

The kubectl top command is an easy way to see node or pod metrics in your terminal.

Use kubectl top node to see the resource consumption of nodes:

$ kubectl top node
NAME                                 CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-agentpool1-26399701-vmss000000   67m          3%     729Mi           10%
k8s-agentpool1-26399701-vmss000001   80m          4%     787Mi           10%
k8s-master-26399701-0                201m         10%    1406Mi          19%

Use kubectl top pod to see the resource consumption of pods:

$ kubectl top pod --namespace=kube-system
NAME                                            CPU(cores)   MEMORY(bytes)
azure-cni-networkmonitor-7gfd4                  2m           15Mi
...
kube-proxy-mzlq5                                1m           18Mi
kube-scheduler-k8s-master-26399701-0            3m           16Mi
metrics-server-bb7db87bc-nm6vn                  1m           12Mi

Kubernetes Dashboard

The Kubernetes Dashboard is a web-based user interface that can visualize cluster metrics.

The Dashboard addon is not enabled by default for an AKS Engine cluster. You must enable it before creating a cluster, as shown in this partial cluster configuration:

"orchestratorProfile": {
  "kubernetesConfig": {
    "addons": [
      {
        "name": "kubernetes-dashboard",
        "enabled": true
      }
    ]
  }

To verify that the Kubernetes Dashboard is running in your cluster:

$ kubectl get pods --namespace=kubernetes-dashboard
NAME                                         READY   STATUS    RESTARTS   AGE
dashboard-metrics-scraper-7bdfbb4477-7mhc8   1/1     Running   0          18h
kubernetes-dashboard-b597987c-rr7bf          1/1     Running   0          18h

Accessing the Dashboard requires a Bearer Token. To create a token for demonstration purposes, you can follow the Kubernetes documentation on creating-a-sample-user.

WARNING: The sample user created in the tutorial above will have administrative privileges and is for demonstration purposes only.

In a terminal window, create a command-line proxy to your cluster with this command:

kubectl proxy

This will make Dashboard available at http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/.

Open that URL in a web browser on the same machine that's running kubectl proxy, and paste your token into the login screen when requested.

After authenticating, you can explore node metrics under the nodes section on the left menu. You can also see pod level metrics under the pods section, and even drill into a specific container in a given pod.

Image of Kubernetes dashboard

For more information about accessing the Dashboard, see the Kubernetes documentation on web UI (dashboard).

Azure Monitor for containers

Azure Monitor for containers is a feature designed to monitor the performance of container workloads. It supports self-managed Kubernetes clusters using AKS Engine.

Azure Monitor for containers gives you performance visibility by collecting memory and processor metrics from controllers, nodes, and containers that are available in Kubernetes through the Metrics API. Container logs are also collected. Please see its documentation overview for more detail.

Azure Monitor for containers is known as the "container-monitoring" addon in AKS Engine, and it is not enabled by default. See the container-monitoring docs to learn how to enable and configure Azure Monitor for containers via addon.

Alternately, you can install the azuremonitor-containers Helm chart to enable Azure Monitor for containers.

Once Azure Monitor for containers is enabled, navigate to https://aka.ms/azmon-containers to view the health, metrics, and logs of your AKS Engine clusters in the Azure portal website.

Refer to azuremonitor-containers-aks-engine for information about supported versions and for troubleshooting.

Image of Azure Monitor for containers

Please reach out with any questions or feedback by sending an email to Azure Monitor for containers.

Monitoring extension

A quick way to scaffold out cloud-native and open source monitoring components is to use the aks-engine monitoring extension. For details on how to use the monitoring extension, please refer to the extension documentation. By embedding the extension in your apimodel, the extension will do much of the work to create a monitoring solution in your cluster, which includes the following:

  • cAdvisor daemon set to publish container metrics
  • Prometheus for metrics collection and storage
  • Grafana for dashboard and visualizations

The extension wires up these components together. Post-deployment of the Kubernetes cluster, you just have to retrieve Grafana admin password (Kubernetes secret) and target your browser to the Grafana endpoint. There is already a pre-loaded Kubernetes cluster monitoring dashboard, so out-of-the-box you will have meaningful monitoring points with the extensibility that Prometheus and Grafana offer you.