[Feature] Elasticagent + conditionally Elasticsearch+Kibana in deployment

[benjamsf]

What

Add in configurations for elasticagent:

• Conditionally, with docker compose profile: kibana, run Elasticsearch and Kibana in the compose, so that you can tinker with logging in eg. a dev deployment.
• If the local setup is not used, then with injected token and url, ship logs to your preferred destination.

How

elastic-agent-config/elastic-agent.yml

agent:
  id: ${ELASTIC_AGENT_ID}

# Fleet enrollment configuration
fleet:
  enabled: ${ELASTIC_AGENT_ENABLED}
  enrollment_token: ${ELASTIC_AGENT_ENROLLMENT_TOKEN}
  hosts:
    - ${ELASTIC_AGENT_FLEET_URL}

# Standalone configuration
outputs:
  default:
    type: elasticsearch
    hosts:
      - ${ELASTICSEARCH_HOSTS:-http://elasticsearch:9200}

inputs:
  - type: docker
    id: docker-logs
    enabled: ${LOG_INPUT_ENABLED:-true}
    streams:
      - containers.ids:
          - '*'  # Collect logs from all containers
    processors:
      - add_docker_metadata: ~
    # Enable the Docker module
    modules:
      - name: docker
        enabled: true

Then in docker-compose.yml
Elasticagent like so:

  elastic-agent:
    image: docker.elastic.co/beats/elastic-agent:8.15.3
    container_name: elastic-agent
    environment:
      ELASTIC_AGENT_ENABLED: "${ELASTIC_AGENT_ENABLED:-false}"
      ELASTIC_AGENT_ENROLLMENT_TOKEN: "${ELASTIC_AGENT_ENROLLMENT_TOKEN}"
      ELASTIC_AGENT_FLEET_URL: "${ELASTIC_AGENT_FLEET_URL}"
      ELASTICSEARCH_HOSTS: "${ELASTICSEARCH_HOSTS:-http://elasticsearch:9200}"
      LOG_INPUT_ENABLED: "${LOG_INPUT_ENABLED:-true}"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup
      - /proc:/hostfs/proc
      - /:/hostfs
      - ./elastic-agent-config/elastic-agent.yml:/usr/share/elastic-agent/elastic-agent.yml
    networks:
      - loggingnet
    restart: unless-stopped

TLDR ELASTIC_AGENT_ENABLED=true, if you want to enroll yourself to a Elastic fleet. If you don't, false, and local elk via using profile: kibana.

Why

Logging is good for you
Tryout

[benjamsf]

We dont really want elk into compose, but built this for the sake of testing logging the containers abit.
Tasks for future:

• config elasticagent for external monitoring
• recommended setup for monitoring