[Flaky Test]: TestFakeComponent – listen tcp 127.0.0.1:6791: bind: address already in use

[rdner]

Failing test case

TestFakeComponent

Error message

listen tcp 127.0.0.1:6791: bind: address already in use

Build

https://buildkite.com/elastic/elastic-agent/builds/8792#018f58b7-68c0-491c-9000-186b7fec221c

OS

Linux

Stacktrace and notes

=== RUN   TestFakeComponent
    fixture.go:281: Extracting artifact elastic-agent-8.14.0-SNAPSHOT-linux-arm64.tar.gz to /tmp/TestFakeComponent2853530698/001
    fixture.go:299: Completed extraction of artifact elastic-agent-8.14.0-SNAPSHOT-linux-arm64.tar.gz to /tmp/TestFakeComponent2853530698/001
    fixture.go:957: All component specifications where removed
    fixture.go:998: Placed component specifications: fake, fake-shipper
    fixture.go:1132: 2024-05-08T15:22:50.584268506Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/elastic-agent/_z5bktXaDsVuiU_BkyqwKhf_LbRssuFQ.sock: connect: no such file or directory" retrying: 147.411399ms
    log.go:85: Flag --disable-encrypted-store has been deprecated, agent on Mac OS can be run/installed without root privileges, see elastic-agent install --help
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:50.706Z","log.origin":{"file.name":"cmd/run.go","file.line":180},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":17516,"agent.version":"8.14.0","agent.unprivileged":true,"ecs.version":"1.6.0"}
    fixture.go:1132: 2024-05-08T15:22:50.732617942Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/elastic-agent/_z5bktXaDsVuiU_BkyqwKhf_LbRssuFQ.sock: connect: no such file or directory" retrying: 176.104264ms
    fixture.go:1132: 2024-05-08T15:22:50.909029849Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/elastic-agent/_z5bktXaDsVuiU_BkyqwKhf_LbRssuFQ.sock: connect: no such file or directory" retrying: 225.462744ms
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:50.970Z","log.origin":{"file.name":"upgrade/rollback.go","file.line":156},"message":"Starting upgrade watcher","log":{"source":"elastic-agent"},"path":"/tmp/TestFakeComponent2853530698/001/elastic-agent-8.14.0-SNAPSHOT-linux-arm64/elastic-agent","args":["/tmp/TestFakeComponent2853530698/001/elastic-agent-8.14.0-SNAPSHOT-linux-arm64/elastic-agent","watch","--path.config","/tmp/TestFakeComponent2853530698/001/elastic-agent-8.14.0-SNAPSHOT-linux-arm64","--path.home","/tmp/TestFakeComponent2853530698/001/elastic-agent-8.14.0-SNAPSHOT-linux-arm64"],"env":[],"dir":"","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:50.970Z","log.origin":{"file.name":"upgrade/rollback.go","file.line":163},"message":"Upgrade Watcher invoked","log":{"source":"elastic-agent"},"agent.upgrade.watcher.process.pid":17522,"agent.process.pid":17516,"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:50.970Z","log.origin":{"file.name":"upgrade/rollback.go","file.line":151},"message":"releasing watcher 17522","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:50.970Z","log.origin":{"file.name":"cmd/run.go","file.line":264},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:50.971Z","log.origin":{"file.name":"application/application.go","file.line":65},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:50.972Z","log.origin":{"file.name":"application/application.go","file.line":71},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"inputs":["fake-isolated-units","fake","fake-apm"],"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:50.972Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":48},"message":"Capabilities file not found in /tmp/TestFakeComponent2853530698/001/elastic-agent-8.14.0-SNAPSHOT-linux-arm64/capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:50.972Z","log.origin":{"file.name":"application/application.go","file.line":77},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    fixture.go:1132: 2024-05-08T15:22:51.135288302Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/elastic-agent/_z5bktXaDsVuiU_BkyqwKhf_LbRssuFQ.sock: connect: no such file or directory" retrying: 247.331825ms
    fixture.go:1132: 2024-05-08T15:22:51.383775347Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/elastic-agent/_z5bktXaDsVuiU_BkyqwKhf_LbRssuFQ.sock: connect: no such file or directory" retrying: 445.426947ms
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:51.607Z","log.origin":{"file.name":"application/application.go","file.line":140},"message":"Elastic Agent has been started in testing mode and is managed through the control protocol","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:51.610Z","log.logger":"control","log.origin":{"file.name":"server/server.go","file.line":88},"message":"GRPC control socket listening at unix:///tmp/elastic-agent/_z5bktXaDsVuiU_BkyqwKhf_LbRssuFQ.sock","log":{"source":"elastic-agent"},"address":"unix:///tmp/elastic-agent/_z5bktXaDsVuiU_BkyqwKhf_LbRssuFQ.sock","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:51.611Z","log.origin":{"file.name":"runtime/manager.go","file.line":213},"message":"Starting grpc control protocol listener on port 6789 with max_message_size 104857600","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:51.613Z","log.logger":"composable.providers.docker","log.origin":{"file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    fixture.go:1132: 2024-05-08T15:22:51.829782294Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/elastic-agent/_z5bktXaDsVuiU_BkyqwKhf_LbRssuFQ.sock: connect: no such file or directory" retrying: 993.990487ms
    fixture.go:1132: 2024-05-08T15:22:52.824004238Z: StateWatch failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /tmp/elastic-agent/_z5bktXaDsVuiU_BkyqwKhf_LbRssuFQ.sock: connect: no such file or directory" retrying: 632.151769ms
    fixture.go:1140: 2024-05-08T15:22:53.456855929Z: StateWatch started
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:53.458Z","log.origin":{"file.name":"application/testing_mode.go","file.line":53},"message":"Testing mode received new configuration from protocol, passing to coordinator","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:53.459Z","log.origin":{"file.name":"upgrade/upgrade.go","file.line":123},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:53.460Z","log.origin":{"file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0x4000835950), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:\"\", APIKey:\"\", SecretToken:\"\", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:53.460Z","log.origin":{"file.name":"monitoring/server.go","file.line":90},"message":"creating monitoring API with cfg api.Config{Enabled:true, Host:\"http://localhost:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:53.460Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:53.460Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:53.475Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":1205},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:54.699Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":608},"message":"Spawned new component fake-default: Starting: spawned pid '17537'","log":{"source":"elastic-agent"},"component":{"id":"fake-default","state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:54.699Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":615},"message":"Spawned new unit fake-default-fake: Starting: spawned pid '17537'","log":{"source":"elastic-agent"},"component":{"id":"fake-default","state":"STARTING"},"unit":{"id":"fake-default-fake","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:54.699Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":615},"message":"Spawned new unit fake-default: Starting: spawned pid '17537'","log":{"source":"elastic-agent"},"component":{"id":"fake-default","state":"STARTING"},"unit":{"id":"fake-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:54.732Z","log.logger":"component.runtime.fake-default","log.origin":{"file.name":"runtime/manager.go","file.line":680},"message":"control checkin v2 protocol has chunking enabled","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:54.733Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":628},"message":"Component state changed fake-default (STARTING->HEALTHY): Healthy: communicating with pid '17537'","log":{"source":"elastic-agent"},"component":{"id":"fake-default","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:54.734Z","message":"unit change received","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-default","type":"fake"},"log":{"source":"fake-default"},"level":"info","state":"STARTING","expectedState":"HEALTHY","time":"2024-05-08T15:22:54Z","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:54.735Z","message":"unit change received","component":{"binary":"fake","dataset":"elastic_agent.fake","id":"fake-default","type":"fake"},"log":{"source":"fake-default"},"level":"info","state":"STARTING","expectedState":"HEALTHY","time":"2024-05-08T15:22:54Z","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:54.736Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":646},"message":"Unit state changed fake-default-fake (STARTING->CONFIGURING): Configuring","log":{"source":"elastic-agent"},"component":{"id":"fake-default","state":"HEALTHY"},"unit":{"id":"fake-default-fake","type":"input","state":"CONFIGURING","old_state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:54.736Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":646},"message":"Unit state changed fake-default (STARTING->CONFIGURING): Configuring","log":{"source":"elastic-agent"},"component":{"id":"fake-default","state":"HEALTHY"},"unit":{"id":"fake-default","type":"output","state":"CONFIGURING","old_state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:54.736Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":646},"message":"Unit state changed fake-default (CONFIGURING->HEALTHY): GRPC fake event pipe connected \"unix:///tmp/elastic-agent/kesXHYHYbU863cFUmh6U9WgsBwPZQPzL.sock\"","log":{"source":"elastic-agent"},"component":{"id":"fake-default","state":"HEALTHY"},"unit":{"id":"fake-default","type":"output","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:55.344Z","log.origin":{"file.name":"application/testing_mode.go","file.line":53},"message":"Testing mode received new configuration from protocol, passing to coordinator","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:55.346Z","log.origin":{"file.name":"upgrade/upgrade.go","file.line":123},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:55.346Z","log.origin":{"file.name":"reload/reload.go","file.line":74},"message":"Stopping monitoring server","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:55.346Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":90},"message":"Stats endpoint (127.0.0.1:6791) finished: accept tcp 127.0.0.1:6791: use of closed network connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:55.347Z","log.origin":{"file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0x400076d380), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:\"\", APIKey:\"\", SecretToken:\"\", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:55.347Z","log.origin":{"file.name":"monitoring/server.go","file.line":90},"message":"creating monitoring API with cfg api.Config{Enabled:true, Host:\"http://localhost:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"error","@timestamp":"2024-05-08T15:22:55.348Z","log.origin":{"file.name":"reload/reload.go","file.line":58},"message":"Failed creating a server: failed to create api server: listen tcp 127.0.0.1:6791: bind: address already in use","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:55.363Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":608},"message":"Spawned new component fake-shipper-default: Starting: spawned pid '17546'","log":{"source":"elastic-agent"},"component":{"id":"fake-shipper-default","state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:55.363Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":615},"message":"Spawned new unit fake-shipper-default: Starting: spawned pid '17546'","log":{"source":"elastic-agent"},"component":{"id":"fake-shipper-default","state":"STARTING"},"unit":{"id":"fake-shipper-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-05-08T15:22:55.363Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":615},"message":"Spawned new unit fake-default: Starting: spawned pid '17546'","log":{"source":"elastic-agent"},"component":{"id":"fake-shipper-default","state":"STARTING"},"unit":{"id":"fake-default","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
    fake_test.go:132: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/fake_test.go:132
        	Error:      	Received unexpected error:
        	            	elastic-agent logged an unexpected error: Failed creating a server: failed to create api server: listen tcp 127.0.0.1:6791: bind: address already in use
        	Test:       	TestFakeComponent
--- FAIL: TestFakeComponent (14.83s)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[belimawr]

I cannot reproduce this one. Looking at the error it looks like there was another Elastic-Agent or Beats running in the same VM, maybe from another test. I'll close this issue for now.