test(TestGenerateProcessorList): close processor before return #47663

kruskall · 2025-11-15T06:22:01Z

Proposed commit message

close the processors to ensure graceful shutdown and avoid goroutine leak.

use logptest logger in test

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

Author's Checklist

[ ]

How to test this PR locally

Use cases

Screenshots

Logs

close the processors to ensure graceful shutdown and avoid goroutine leak. use logptest logger in test

github-actions · 2025-11-15T06:22:11Z

🤖 GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2025-11-15T06:22:37Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kruskall? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

elasticmachine · 2025-11-16T14:08:44Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

belimawr

I'm still seeing some failures due to testing.T being used after the test has ended:

% git log -1                                       
commit 947320cd4d1e281c9b2811876d65b6bf67da87e1 (HEAD, kruskall/test/TestGenerateProcessorList-flaky)
Merge: d067abef80 c2fc36f800
Author: kruskall <99559985+kruskall@users.noreply.github.com>
Date:   Sat Nov 22 15:08:20 2025 +0100

    Merge branch 'main' into test/TestGenerateProcessorList-flaky
tiago@millennium-falcon beats/libbeat/publisher/processing  v1.25.1  HEAD [$?] 
% go test -count=500 -run=TestGenerateProcessorList
panic: Log in goroutine after TestGenerateProcessorList has completed: 2025-11-24T09:40:43.746-0500     DEBUG   add_cloud_metadata      add_cloud_metadata: received disposition for azure after 4.235739ms. result=[provider:azure, error=failed requesting azure metadata: Get "http://169.254.169.254/metadata/instance/compute?api-version=2021-02-01": dial tcp 169.254.169.254:80: connect: network is unreachable, metadata={}]
        
        panic: Log in goroutine after TestGenerateProcessorList has completed: 2025-11-24T09:40:43.746-0500     DEBUG   add_cloud_metadata      add_cloud_metadata: fetchMetadata ran for 4.263064ms
        

goroutine 543 [running]:
testing.(*common).log(0xc000a90fc0, {0xc000d29260?, 0x6a?})
        /usr/local/go/src/testing/testing.go:1030 +0x1ac
testing.(*common).Logf(0xc000a90fc0, {0x525cab3?, 0x400?}, {0xc000e00700?, 0xff?, 0x2?})
        /usr/local/go/src/testing/testing.go:1191 +0x4f
go.uber.org/zap/zaptest.TestingWriter.Write({{0x7f797875cd20?, 0xc000a90fc0?}, 0xc0?}, {0xc000995000?, 0x6b, 0xc000995000?})
        /home/tiago/go/pkg/mod/go.uber.org/zap@v1.27.0/zaptest/logger.go:146 +0xdb
go.uber.org/zap/zapcore.(*ioCore).Write(0xc0006cff20, {0xff, {0xc241397aec8374ef, 0x439011c, 0x7552d00}, {0x527f9b6, 0x12}, {0xc000540540, 0x34}, {0x0, ...}, ...}, ...)
        /home/tiago/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0xb5
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000cbe750, {0x0, 0x0, 0x0})
        /home/tiago/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x119
go.uber.org/zap.(*SugaredLogger).log(0xc000cb2148, 0xff, {0x52ccb0d?, 0x2c?}, {0xc000671680?, 0x1?, 0x1?}, {0x0, 0x0, 0x0})
        /home/tiago/go/pkg/mod/go.uber.org/zap@v1.27.0/sugar.go:355 +0xec
go.uber.org/zap.(*SugaredLogger).Debugf(...)
        /home/tiago/go/pkg/mod/go.uber.org/zap@v1.27.0/sugar.go:198
github.com/elastic/elastic-agent-libs/logp.(*Logger).Debugf(...)
        /home/tiago/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.26.0/logp/logger.go:227
github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata.func1()
        /home/tiago/devel/beats/libbeat/processors/add_cloud_metadata/providers.go:160 +0x91
panic({0x4af3f80?, 0xc000e006c0?})
        /usr/local/go/src/runtime/panic.go:783 +0x132
testing.(*common).log(0xc000a90fc0, {0xc000112c60?, 0x154?})
        /usr/local/go/src/testing/testing.go:1030 +0x1ac
testing.(*common).Logf(0xc000a90fc0, {0x525cab3?, 0x400?}, {0xc000e006a0?, 0xff?, 0x2?})
        /usr/local/go/src/testing/testing.go:1191 +0x4f
go.uber.org/zap/zaptest.TestingWriter.Write({{0x7f797875cd20?, 0xc000a90fc0?}, 0xc0?}, {0xc0006af000?, 0x155, 0xc0006af000?})
        /home/tiago/go/pkg/mod/go.uber.org/zap@v1.27.0/zaptest/logger.go:146 +0xdb
go.uber.org/zap/zapcore.(*ioCore).Write(0xc0006cff20, {0xff, {0xc241397aec8349a3, 0x438d5d2, 0x7552d00}, {0x527f9b6, 0x12}, {0xc000a18480, 0x11e}, {0x0, ...}, ...}, ...)
        /home/tiago/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0xb5
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00080ac30, {0x0, 0x0, 0x0})
        /home/tiago/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x119
go.uber.org/zap.(*SugaredLogger).log(0xc000cb2148, 0xff, {0x52fcb88?, 0x3c?}, {0xc000671c68?, 0x3?, 0x3?}, {0x0, 0x0, 0x0})
        /home/tiago/go/pkg/mod/go.uber.org/zap@v1.27.0/sugar.go:355 +0xec
go.uber.org/zap.(*SugaredLogger).Debugf(...)
        /home/tiago/go/pkg/mod/go.uber.org/zap@v1.27.0/sugar.go:198
github.com/elastic/elastic-agent-libs/logp.(*Logger).Debugf(...)
        /home/tiago/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.26.0/logp/logger.go:227
github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.acceptFirstPriorityResult({0x5827df0, 0xc00028e310}, 0xc00071a6d8, {0xc000cb2148?, 0xc00071a3ff?, 0x7552d00?}, 0xc00078ddc0)
        /home/tiago/devel/beats/libbeat/processors/add_cloud_metadata/providers.go:206 +0x29d
github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata(0xc00040ede0)
        /home/tiago/devel/beats/libbeat/processors/add_cloud_metadata/providers.go:191 +0x445
github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1()
        /home/tiago/devel/beats/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go:101 +0x67
sync.(*Once).doSlow(0xc000a90fc0?, 0xc0009b2f90?)
        /usr/local/go/src/sync/once.go:78 +0xac
sync.(*Once).Do(...)
        /usr/local/go/src/sync/once.go:69
github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init(0x53adb50?)
        /home/tiago/devel/beats/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go:99 +0x31
created by github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.New in goroutine 542
        /home/tiago/devel/beats/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go:89 +0x2ae
exit status 2
FAIL    github.com/elastic/beats/v7/libbeat/publisher/processing        0.080s
tiago@millennium-falcon beats/libbeat/publisher/processing  v1.25.1  HEAD [$?] 
%

belimawr · 2025-11-24T14:42:33Z

I accidentally dismissed @mauri870's review 🙈. Sorry for that.

mauri870 · 2025-11-24T14:51:05Z

I'm still seeing some failures due to testing.T being used after the test has ended:

Interesting, I did test it with ./script/stresstest.sh ./libbeat/publisher/processing ^TestGenerateProcessorList$ -p 128 and it reported no errors even after 40k runs. I guess this one only manifests when we run the test multiple times within the go test process.

kruskall · 2025-11-24T15:01:36Z

I'm still seeing some failures due to testing.T being used after the test has ended:

Yup, unfortunately there are two required PRs before the flaky test is fully fixed (see description): #47661 and #47662

test(TestGenerateProcessorList): close processor before return

d067abe

close the processors to ensure graceful shutdown and avoid goroutine leak. use logptest logger in test

kruskall requested a review from a team as a code owner November 15, 2025 06:22

kruskall requested review from AndersonQ and VihasMakwana November 15, 2025 06:22

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 15, 2025

mergify bot assigned kruskall Nov 15, 2025

kruskall added the flaky-test Unstable or unreliable test cases. label Nov 15, 2025

pierrehilbert added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Nov 16, 2025

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 16, 2025

pierrehilbert added the skip-changelog label Nov 16, 2025

Merge branch 'main' into test/TestGenerateProcessorList-flaky

947320c

kruskall mentioned this pull request Nov 22, 2025

[Flaky test] TestGenerateProcessorList: panic: Log in goroutine after TestGenerateProcessorList has completed #47528

Open

pierrehilbert requested review from belimawr and removed request for AndersonQ November 23, 2025 16:08

mauri870 approved these changes Nov 24, 2025

View reviewed changes

belimawr approved these changes Nov 24, 2025

View reviewed changes

belimawr reviewed Nov 24, 2025

View reviewed changes

belimawr requested review from belimawr and mauri870 November 24, 2025 14:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(TestGenerateProcessorList): close processor before return #47663

test(TestGenerateProcessorList): close processor before return #47663

kruskall commented Nov 15, 2025

Uh oh!

github-actions bot commented Nov 15, 2025

Uh oh!

mergify bot commented Nov 15, 2025

Uh oh!

elasticmachine commented Nov 16, 2025

Uh oh!

belimawr left a comment •

edited

Loading

Uh oh!

belimawr commented Nov 24, 2025

Uh oh!

mauri870 commented Nov 24, 2025 •

edited

Loading

Uh oh!

kruskall commented Nov 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

test(TestGenerateProcessorList): close processor before return #47663

Are you sure you want to change the base?

test(TestGenerateProcessorList): close processor before return #47663

Conversation

kruskall commented Nov 15, 2025

Proposed commit message

Checklist

Disruptive User Impact

Author's Checklist

How to test this PR locally

Use cases

Screenshots

Logs

Uh oh!

github-actions bot commented Nov 15, 2025

🤖 GitHub comments

Uh oh!

mergify bot commented Nov 15, 2025

Uh oh!

elasticmachine commented Nov 16, 2025

Uh oh!

belimawr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

belimawr commented Nov 24, 2025

Uh oh!

mauri870 commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kruskall commented Nov 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

belimawr left a comment •

edited

Loading

mauri870 commented Nov 24, 2025 •

edited

Loading