Agent service doesn't get started and host not available under Agents tab on reboot when installed with --delay-enroll.

[amolnater-qasource]

Kibana Build details:

VERSION: 8.12.0 BC3
BUILD: 69985
COMMIT: 2a8afed8572a4c709aa1c64216748197eeb9b18f
Artifact Link: https://staging.elastic.co/8.12.0-61156bc6/summary-8.12.0.html

Host OS: Windows French, Inconsistently on Korean

Preconditions:

1. 8.12.0 BC3 Kibana cloud environment should be available.

Steps to reproduce:

1. Run agent install command with --delay-enroll on Windows 10 French.
2. Restart the VM and re-access the VM.
3. Observe host is not available under Agents tab and agent service is not restarted on reboot.

NOTE:

• Issue is also inconsistently reproducible on Korean VM.

Screenshot:

Expected Result:
Agent service should get started and host should be available under Agents tab on reboot when installed with --delay-enroll.

Previously observed:
#2719

Logs:
No agent logs are available.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[cmacknz]

Does this happen with an English VM? If not I assume the root cause will turn out to be similar to #3960.

[cmacknz]

Can you collect agent diagnostics? If you can't, can you attach a zip archive of the agent installation directory?

There should be logs explaining why the delayed enrollment failed.

[amolnater-qasource]

Hi @cmacknz

Thank you for looking into this.

We have revalidated this issue on latest 8.12.0 BC4 and found it still reproducible.

We are not able to fetch the logs for this agent:

Hence, we have shared the agent directory zip over slack.

Does this happen with an English VM?

The issue is not reproducible on English VM.

Build details:
VERSION: 8.12.0 BC4
BUILD: 70016
COMMIT: c2fda4713eb89786cf07dba596f6d45136858fd5

Please let us know if anything else is required from our end.

Thanks!!

[cmacknz]

Thanks, the agent directory zip unfortunately didn't contain any logs.

My assumption is that this is the same problem as #3960. Let's retest once that issue is resolved to confirm.

[pierrehilbert]

@amolnater-qasource could you please test this again now that we have BC5 with @leehinman 's fix?

[amolnater-qasource]

Hi Team,

We have revalidated this issue on latest 8.12.0 BC5 kibana cloud environment and found it still reproducible.

Observations:

• Agent service doesn't get started and host gets stuck under Updating state under Agents tab on reboot when installed with --delay-enroll.
  • Reproduced with Windows French and German VMs twice on BC5 .

Build details:
VERSION: 8.12.0
BUILD: 70053
COMMIT: db9b8921b37139cbb1e11d23f6381f655edeb72b
Artifact Link: https://staging.elastic.co/8.12.0-9f05a310/downloads/beats/elastic-agent/elastic-agent-8.12.0-windows-x86_64.zip

Screenshot:

Few logs are generated under agent installation directory, we have shared the same under #elastic-agent-team channel over slack.

UPDATE:
Added files at below links:

curl -L -H 'Authorization: a62f423df4e6405e' -o 'Agent2.zip' https://upload.elastic.co/d/568ecf27cd98088ee7b6aa88da0219127a1be68cf1fc6f56c0c9752893832b37

curl -L -H 'Authorization: be86d0fc5d76c7e4' -o 'Agent.zip' https://upload.elastic.co/d/80d4bb5413339320a4b8476bb12d525a14d1e147c58ff6229639614b01f5d337

Please let us know if we are missing anything here.

Thanks!

[cmacknz]

https://github.com/elastic/elastic-agent/commits/db9b8921b37139cbb1e11d23f6381f655edeb72b isn't a valid elastic-agent commit.

Downloading that version and checking the version manually I get:

./elastic-agent version
Binary: 8.12.0 (build: 8744ca087b3908495b0204e6c80dd9b4de92ccf5 at 2024-01-05 15:39:18 +0000 UTC)
Daemon: <failed to communicate>
could not get version. failed to communicate with running daemon: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/elastic-agent/e0facba31ae0abb8bd29c6a87a85829cf8725aba71a45169d6e7fb61272d3c09.sock: connect: no such file or directory"
Use --binary-only flag to skip trying to retrieve version from running daemon

That is new enough to have the fix c02f65a

[cmacknz]

Errors from the first set of logs:

{"log.level":"info","@timestamp":"2024-01-08T08:43:29.186Z","log.origin":{"file.name":"cmd/run.go","file.line":157},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":3696,"agent.version":"8.12.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T08:43:29.801Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Starting enrollment to URL: https://bf5d27ec208440a4b53d82c8e9d9b005.fleet.europe-west1.gcp.cloud.es.io:443/","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T08:43:31.332Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 0","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:43:31.610Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 0 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Das System kann die angegebene Datei nicht finden.\"'. Waiting for 2s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T08:43:33.617Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 1","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:43:33.628Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 1 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Das System kann die angegebene Datei nicht finden.\"'. Waiting for 4s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T08:43:37.635Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 2","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:43:37.636Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 2 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Das System kann die angegebene Datei nicht finden.\"'. Waiting for 8s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T08:43:45.637Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 3","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:43:45.638Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 3 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Das System kann die angegebene Datei nicht finden.\"'. Waiting for 16s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T08:44:01.652Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 4","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:44:01.653Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 4 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Das System kann die angegebene Datei nicht finden.\"'. Waiting for 32s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:44:33.666Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":281},"message":"Elastic Agent might not be running; unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Das System kann die angegebene Datei nicht finden.\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:44:33.692Z","log.origin":{"file.name":"cmd/run.go","file.line":162},"message":"failed to perform delayed enrollment: could not reload agent daemon, unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Das System kann die angegebene Datei nicht finden.\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

Logs from the second agent are the same:

{"log.level":"info","@timestamp":"2024-01-08T08:43:13.801Z","log.origin":{"file.name":"cmd/run.go","file.line":157},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":3412,"agent.version":"8.12.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T08:43:14.275Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Starting enrollment to URL: https://bf5d27ec208440a4b53d82c8e9d9b005.fleet.europe-west1.gcp.cloud.es.io:443/","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T08:43:15.814Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 0","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:43:15.875Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 0 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Le fichier spécifié est introuvable.\"'. Waiting for 2s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T08:43:17.878Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 1","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:43:17.878Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 1 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Le fichier spécifié est introuvable.\"'. Waiting for 4s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T08:43:21.885Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 2","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:43:21.885Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 2 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Le fichier spécifié est introuvable.\"'. Waiting for 8s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T08:43:29.897Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 3","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:43:29.898Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 3 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Le fichier spécifié est introuvable.\"'. Waiting for 16s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T08:43:45.907Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 4","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:43:45.908Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 4 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Le fichier spécifié est introuvable.\"'. Waiting for 32s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:44:17.913Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":281},"message":"Elastic Agent might not be running; unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Le fichier spécifié est introuvable.\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T08:44:17.931Z","log.origin":{"file.name":"cmd/run.go","file.line":162},"message":"failed to perform delayed enrollment: could not reload agent daemon, unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: open \\\\\\\\.\\\\pipe\\\\elastic-agent-system: Le fichier spécifié est introuvable.\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

[cmacknz]

I can reproduce this on an english Linux Debian 11 VM, I don't think this is related to the language:

craig_mackenzie@cmackenzie-debian11-test:~$ sudo elastic-agent version
Binary: 8.12.0 (build: 8744ca087b3908495b0204e6c80dd9b4de92ccf5 at 2024-01-05 15:38:21 +0000 UTC)
Daemon: <failed to communicate>
could not get version. failed to communicate with running daemon: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/elastic-agent.sock: connect: no such file or directory"
Use --binary-only flag to skip trying to retrieve version from running daemon
craig_mackenzie@cmackenzie-debian11-test:~$ sudo cat /opt/Elastic/Agent/data/elastic-agent-8744ca/logs/elastic-agent-20240108.ndjson
{"log.level":"info","@timestamp":"2024-01-08T15:23:40.919Z","log.origin":{"file.name":"cmd/run.go","file.line":157},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":355,"agent.version":"8.12.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T15:23:41.578Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Starting enrollment to URL: https://2e6c294352624beda2f6830fada574fb.fleet.eastus2.staging.azure.foundit.no:443/","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T15:23:43.122Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 0","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T15:23:43.131Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 0 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 2s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T15:23:45.132Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 1","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T15:23:45.133Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 1 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 4s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T15:23:49.135Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 2","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T15:23:49.136Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 2 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 8s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T15:23:57.138Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 3","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T15:23:57.139Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 3 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 16s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-01-08T15:24:13.139Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 4","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T15:24:13.140Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Restart attempt 4 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 32s","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T15:24:45.140Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":281},"message":"Elastic Agent might not be running; unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/elastic-agent.sock: connect: no such file or directory\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-01-08T15:24:45.140Z","log.origin":{"file.name":"cmd/run.go","file.line":162},"message":"failed to perform delayed enrollment: could not reload agent daemon, unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/elastic-agent.sock: connect: no such file or directory\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

[amolnater-qasource]

Hi @cmacknz

The commits for the BC5 agent version are post merge, for the agent which we have tested and the issue was reproducible:

Artifact: https://staging.elastic.co/8.12.0-9f05a310/downloads/beats/elastic-agent/elastic-agent-8.12.0-windows-x86_64.zip

Below commits are of the kibana BC5 build:

BUILD: 70053
COMMIT: db9b8921b37139cbb1e11d23f6381f655edeb72b

Could you please confirm if both kibana and agent versions should be post the merge or if we are missing anything here?

Thanks

[cmacknz]

Only the agent needs to be post merge

[amolnater-qasource]

Hi Team,

We have revalidated this issue on latest 8.12.0 BC6 kibana cloud environment and found it fixed now.

Observations:

• Agent service get started and host is available under Agents tab on reboot when installed with --delay-enroll.

Host:
French-Windows-10

Build details:
VERSION: 8.12.0 BC6
BUILD: 70088
COMMIT: e9092c0a17923f4ed984456b8a5db619b0a794b3
Artifact Link: https://staging.elastic.co/8.12.0-3eba7f46/summary-8.12.0.html

Screenshot:

Hence we are marking this issue as QA:Validated.

Thanks!