TestInitDatabaseService/enabled_invalid_databases flakiness

[ravicious]

Failure

Link(s) to logs

• https://github.com/gravitational/teleport/actions/runs/9514256517/job/26226076225#step:6:1579
• https://github.com/gravitational/teleport/actions/runs/9514855486/job/26227882953#step:6:1578

Relevant snippet

    service_test.go:1807: 
        	Error Trace:	/__w/teleport/teleport/lib/service/service_test.go:1807
        	Error:      	Received unexpected error:
        	            	context deadline exceeded
        	Test:       	TestInitDatabaseService/enabled_invalid_databases

[ibeckermayer]

Another hit on master: https://github.com/gravitational/teleport/actions/runs/9552074016/job/26327820702?pr=42277

[ibeckermayer]

https://github.com/gravitational/teleport/actions/runs/9554888096/job/26336908713

[zmb3]

https://github.com/gravitational/teleport/actions/runs/9568713659/job/26379558815

https://github.com/gravitational/teleport/actions/runs/9568930078/job/26380268155

[zmb3]

New hit here: https://github.com/gravitational/teleport/actions/runs/10798727879/job/29952841726?pr=46226

Looks like the tempdir cleanup problem:

=== FAIL: lib/service TestInitDatabaseService/enabled_invalid_databases (6.21s)
    testing.go:1232: TempDir RemoveAll cleanup: unlinkat /tmp/TestInitDatabaseServiceenabled_invalid_databases626223594/003: directory not empty

[rosstimothy]

Nightly test runner is also hitting the directory cleanup:

OUTPUT github.com/gravitational/teleport/lib/service.TestInitDatabaseService
===================================================
=== RUN   TestInitDatabaseService
=== PAUSE TestInitDatabaseService
=== CONT  TestInitDatabaseService
{"timestamp":"2024-10-14T11:14:55Z","level":"debug","caller":"lite/lite.go:259","message":"Connected to database","component":"sqlite","database":"file:/tmp/847634349/proc/sqlite.db?_busy_timeout=10000&_sync=FULL&_txlock=immediate","poll_stream_period":1000000000}
--- FAIL: TestInitDatabaseService (0.00s)
{"timestamp":"2024-10-14T11:14:58Z","level":"error","caller":"machineidv1/spiffe_federation_syncer.go:187","message":"SPIFFEFederation syncer encountered a fatal error, it will restart after backoff","component":"proc:12","pid":"552672.12","component":"auth:12","component":"auth:spiffe_federation_syncer","error":"disk I/O error: no such file or directory","restart_after":8760519528}
{"caller":"service/supervisor.go:255","component":"proc:12","level":"debug","message":"Service is completed and removed.","pid":"552672.12","service":"auth.spiffe_federation_syncer","timestamp":"2024-10-14T11:14:58Z"}
===================================================
OUTPUT github.com/gravitational/teleport/lib/service.TestInitDatabaseService/enabled_invalid_databases
===================================================
=== RUN   TestInitDatabaseService/enabled_invalid_databases
=== PAUSE TestInitDatabaseService/enabled_invalid_databases
=== CONT  TestInitDatabaseService/enabled_invalid_databases
    testing.go:1232: TempDir RemoveAll cleanup: unlinkat /tmp/TestInitDatabaseServiceenabled_invalid_databases2806506600/001: directory not empty
--- FAIL: TestInitDatabaseService/enabled_invalid_databases (3.05s)```

[rosstimothy]

This test is likely prone to the same issues that TestDynamicClientReuse had. Namely, we are closing the process without waiting for all the services to be completed first.

[espadolini]

Just hit this again even after #49477 😭

https://github.com/gravitational/teleport/actions/runs/12415550001/job/34662170201