Monitoring

Setup

Note about environment variables

If you use the below config to set "METRICS_ENABLED" to 'true' inside of docker-compose.override.yml, you will need to edit env/netbox.env and comment out the METRICS_ENABLED line there. This is (I think) because the environment file is sourced after the docker-compose override. I think this line should probably be removed from env/netbox.env to reduce confusion, since it's the default.

Docker image tags

The prometheus and grafana docker images are set to use the "latest" version. You might want to pin these to specific versions in your own deployment.

---

Add the following to your docker-compose.override.yml (or create that file if you haven't done so):

services:
  # netbox
  netbox:
    environment:
      METRICS_ENABLED: 'true'
      prometheus_multiproc_dir: /tmp/metrics
    volumes:
    - type: tmpfs
      target: /tmp/metrics
      read_only: false
      tmpfs:
        mode: 0o01777

  # postgres
  postgres-exporter:
    image: wrouesnel/postgres_exporter:v0.8.0
    depends_on:
    - postgres
    environment:
      DATA_SOURCE_URI: postgres?sslmode=disable
      DATA_SOURCE_USER: netbox
      DATA_SOURCE_PASS: J5brHrAXFLQSif0K
      PG_EXPORTER_AUTO_DISCOVER_DATABASES: 'true'

  # redis
  redis-worker-exporter:
    image: oliver006/redis_exporter
    depends_on:
    - redis
    environment:
      REDIS_ADDR: redis://redis:6379
      REDIS_PASSWORD: H733Kdjndks81
  redis-cache-exporter:
    image: oliver006/redis_exporter
    depends_on:
    - redis-cache
    environment:
      REDIS_ADDR: redis://redis-cache:6379
      REDIS_PASSWORD: t4Ph722qJ5QHeQ1qfu36

  # prometheus
  prometheus:
    image: prom/prometheus:latest
    depends_on:
    - postgres-exporter
    - redis-cache-exporter
    - redis-worker-exporter
    - netbox
    ports:
    - '9090:9090'
    volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml
    - prometheus-data:/prometheus/data

  # grafana
  grafana:
    image: grafana/grafana:latest
    depends_on:
    - prometheus
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: admin
      GF_SECURITY_SECRET_KEY: oew5RCBGGBba0MdsEKrj
      GF_METRICS_ENABLED: 'true'
    ports:
    - '3000:3000'
    volumes:
    #- ./monitoring/grafana/plugins/:/var/lib/grafana/plugins/:z,ro
    #- ./monitoring/grafana/provisioning/:/etc/grafana/provisioning/:z,ro
    #- ./monitoring/grafana/dashboards/:/etc/grafana/dashboards/:z,ro
    - grafana-data:/var/lib/grafana

volumes:
  prometheus-data:
    driver: local
  grafana-data:
    driver: local

Then create the new file prometheus.yml:

scrape_configs:
- job_name: prometheus
  static_configs:
  - targets: ['localhost:9090']

- job_name: netbox
  static_configs:
  - targets: ['netbox:8080']
    labels:
      app: 'netbox'

- job_name: postgresql
  static_configs:
  - targets: ['postgres-exporter:9187']

- job_name: redis
  static_configs:
  - targets: ['redis-worker-exporter:9121', 'redis-cache-exporter:9121']

- job_name: grafana
  static_configs:
  - targets: ['grafana:3000']

Then run docker-compose up.

Prometheus

Prometheus grabs the data once every minute from each service (NetBox, Nginx, PostgreSQL, 2x Redis, Prometheus itself) and stores it in a TSDB. You can access Prometheus on port 9090.

Grafana

Grafana provides visualization of the data, which can be composed into dashboard. To fetch the data it talks directly to the Prometheus service. You can access Grafana on port 3000. The default credentials are admin:admin.

You will find several user-made dashboards on the Grafana page which are easy to import using the Grafana Dashboard ID. There are already dashboard for Nginx, PostgreSQL, Redis, Prometheus and also Django (which is the framework on which NetBox is built). For more information see the importing a dashboard and discover dashboards on grafana.com articles.

https://grafana.com/grafana/dashboards/17658-django/ works well with django-prometheus. It needs "app: 'netbox'" assigned in the labels which I have added to the example prometheus.yml above.

The "Cache Hit Ratio" does not work because netbox is not exposing it's cache information through django. According to (https://github.com/netbox-community/netbox/issues/13349) nothing is really cached directly, so they don't plan to add these.