Skip to content

Commit

Permalink
ingest: fix flaky TestConcurrentFetchers (#9926)
Browse files Browse the repository at this point in the history
There's a race between produceCtx being cancelled and producing the record with it.

This is the error that I saw in CI

```
--- FAIL: TestConcurrentFetchers (1.22s)
    --- FAIL: TestConcurrentFetchers/update_concurrency_with_continuous_production (10.01s)
        logging.go:33: level info component kafka_client msg immediate metadata update triggered why from user ForceMetadataRefresh
        logging.go:33: level info component kafka_client msg producing to a new topic for the first time, fetching metadata to learn its partitions topic test-topic
        logging.go:33: level info component kafka_client msg immediate metadata update triggered why forced load because we are producing to a topic for the first time
        logging.go:33: level info component kafka_client msg done waiting for metadata for new topic topic test-topic
        logging.go:33: level info component kafka_client msg initializing producer id
        logging.go:33: level info component kafka_client msg producer id initialization success id 2916317827932012471 epoch 0
        reader_test.go:2339:
            	Error Trace:	/__w/mimir/mimir/pkg/storage/ingest/reader_test.go:2339
            	            				/__w/mimir/mimir/pkg/storage/ingest/fetcher_test.go:650
            	            				/usr/local/go/src/runtime/asm_amd64.s:1700
            	Error:      	Received unexpected error:
            	            	context canceled
            	Test:       	TestConcurrentFetchers/update_concurrency_with_continuous_production
        fetcher_test.go:692:
            	Error Trace:	/__w/mimir/mimir/pkg/storage/ingest/fetcher_test.go:692
            	Error:      	Not equal:
            	            	expected: 782
            	            	actual  : 783
            	Test:       	TestConcurrentFetchers/update_concurrency_with_continuous_production
            	Messages:   	Should not fetch more records than produced
        fetcher_test.go:706: Total produced: 783, Total fetched: 782
        fetcher_test.go:707: Fetched with initial concurrency: 201
        fetcher_test.go:708: Fetched with high concurrency: 290
        fetcher_test.go:709: Fetched with low concurrency: 291
FAIL
```

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
  • Loading branch information
dimitarvdimitrov authored Nov 18, 2024
1 parent 92cc594 commit 60e15f1
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion pkg/storage/ingest/fetcher_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -647,7 +647,8 @@ func TestConcurrentFetchers(t *testing.T) {
case <-ticker.C:
count := producedCount.Inc()
record := fmt.Sprintf("record-%d", count)
produceRecord(produceCtx, t, client, topicName, partitionID, []byte(record))
// Use context.Background() so that we don't race with the test context being cancelled.
produceRecord(context.Background(), t, client, topicName, partitionID, []byte(record))
}
}
}()
Expand Down

0 comments on commit 60e15f1

Please sign in to comment.