Docker logs are not sent to loki after a docker restart

[commarla]

What's wrong?

Hi,

I have a simple 2 containers setup. The first one is Alloy called grafana-alloy and the 2nd is one my application called daemon.
Alloy is configured with docker discovery and loki receiver. Everything work fine until I docker restart daemon. Logs are not send anymore until I docker restart grafana-alloy.

Steps to reproduce

I start Alloy with this command:

docker run --name grafana-alloy --net=host --pid=host --user root \
      -v "$HOME/certs/cert.pem:/mnt/config/cert.pem" \
      -v "$HOME/certs/key.pem:/mnt/config/key.pem" \
      -v "$HOME/grafana-config/grafana-config.alloy:/mnt/config/config.alloy" \
      -v "$HOME/grafana-data/:/mnt/data" \
      -v /var/run:/var/run:ro\
      -v /sys:/sys:ro \
      -v /:/host:ro \
      -v /var/lib/docker/:/var/lib/docker \
      -v /var/run/docker.sock:/var/run/docker.sock:ro \
      grafana/alloy:v1.0.0 run --server.http.listen-addr=0.0.0.0:12345 /mnt/config/config.alloy --storage.path=/mnt/data

And just do a docker restart daemon which is a simple go app in a container.

In the log we can see the log collection stopped and when I restart alloy the process loop starting again.

System information

Ubuntu 22.04.3 LTS arm64

Software version

Grafana Alloy 1.0.0 docker 25.0.4

Configuration

logging {
  level  = "debug"
}

discovery.docker "containers" {
	host = "unix:///var/run/docker.sock"
}

discovery.relabel "containers" {
	targets = discovery.docker.containers.targets

	rule {
		action        = "replace"
		source_labels = ["__meta_docker_container_name"]
		target_label  = "container_name"
	}

	rule {
		action        = "replace"
		source_labels = ["__meta_docker_container_id"]
		target_label  = "container_id"
	}

	rule {
		action        = "replace"
		source_labels = ["__meta_docker_container_id"]
		target_label  = "container_id"
	}
}

loki.source.docker "containers_logs" {
	host       = "unix:///var/run/docker.sock"
	targets    = discovery.relabel.containers.output
	forward_to = [loki.relabel.add_labels.receiver]
}

loki.relabel "add_labels" {
	forward_to = [loki.write.default.receiver]

	rule {
		target_label = "cell_hostname"
		replacement  = "vm"
	}

	rule {
		target_label = "cell_category"
		replacement  = "unknown"
	}

	rule {
		target_label = "cell_location"
		replacement  = "unknown"
	}
}

loki.write "default" {
	endpoint {
		url = "https://<REDACTED>/loki/api/v1/push"

		tls_config {
			cert_file = "/mnt/config/cert.pem"
			key_file  = "/mnt/config/key.pem"
		}
	}
}

Logs

ts=2024-04-26T16:18:29.902235976Z level=debug msg="starting process loop" component_path=/ component_id=loki.source.docker.containers_logs target=docker/13211fb740f5310e3879c59f02da92458680b0f6df7e621155f8ddb24c6369a8 container=13211fb740f5310e3879c59f02da92458680b0f6df7e621155f8ddb24c6369a8

// DOCKER RESTART DAEMON 

ts=2024-04-26T16:18:54.12623348Z level=info msg="finished transferring logs" component_path=/ component_id=loki.source.docker.containers_logs target=docker/13211fb740f5310e3879c59f02da92458680b0f6df7e621155f8ddb24c6369a8 written=364 container=13211fb740f5310e3879c59f02da92458680b0f6df7e621155f8ddb24c6369a8
ts=2024-04-26T16:18:54.126312814Z level=debug msg="done processing Docker logs" component_path=/ component_id=loki.source.docker.containers_logs target=docker/13211fb740f5310e3879c59f02da92458680b0f6df7e621155f8ddb24c6369a8 container=13211fb740f5310e3879c59f02da92458680b0f6df7e621155f8ddb24c6369a8
ts=2024-04-26T16:18:54.126364398Z level=debug msg="stopped Docker target" component_path=/ component_id=loki.source.docker.containers_logs target=docker/13211fb740f5310e3879c59f02da92458680b0f6df7e621155f8ddb24c6369a8 container=13211fb740f5310e3879c59f02da92458680b0f6df7e621155f8ddb24c6369a8
ts=2024-04-26T16:18:54.311533127Z level=debug msg="stopped Docker target" component_path=/ component_id=loki.source.docker.containers_logs target=docker/13211fb740f5310e3879c59f02da92458680b0f6df7e621155f8ddb24c6369a8 container=13211fb740f5310e3879c59f02da92458680b0f6df7e621155f8ddb24c6369a8


///// HERE IS WHEN I RESTART ALLOY


ts=2024-04-26T16:19:27.714682566Z level=debug msg="starting process loop" component_path=/ component_id=loki.source.docker.containers_logs target=docker/13211fb740f5310e3879c59f02da92458680b0f6df7e621155f8ddb24c6369a8 container=13211fb740f5310e3879c59f02da92458680b0f6df7e621155f8ddb24c6369a8

[wildum]

Thanks for reporting this bug! I found out that there is already an issue for this bug so I'm gonna close this ticket and continue the investigation in the other ticket: #278