Skip to content

Commit

Permalink
fix: noisy broker subscription failed error during storage broker d…
Browse files Browse the repository at this point in the history
…eploys (#10521)

During broker deploys, pageservers log this noisy WARN en masse.

I can trivially reproduce the WARN message in neon_local by SIGKILLing
broker during e.g. `pgbench -i`.

I don't understand why tonic is not detecting the error as
`Code::Unavailable`.

Until we find time to understand that / fix upstream, this PR adds the
error message to the existing list of known error messages that get
demoted to INFO level.

Refs:
-  refs #9562
  • Loading branch information
problame authored Jan 27, 2025
1 parent ebf4421 commit 3d36dfe
Showing 1 changed file with 2 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -164,9 +164,10 @@ pub(super) async fn connection_manager_loop_step(
Ok(Some(broker_update)) => connection_manager_state.register_timeline_update(broker_update),
Err(status) => {
match status.code() {
Code::Unknown if status.message().contains("stream closed because of a broken pipe") || status.message().contains("connection reset") => {
Code::Unknown if status.message().contains("stream closed because of a broken pipe") || status.message().contains("connection reset") || status.message().contains("error reading a body from connection") => {
// tonic's error handling doesn't provide a clear code for disconnections: we get
// "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe"
// => https://github.com/neondatabase/neon/issues/9562
info!("broker disconnected: {status}");
},
_ => {
Expand Down

1 comment on commit 3d36dfe

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

7554 tests run: 7156 passed, 1 failed, 397 skipped (full report)


Failures on Postgres 16

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_layer_map[release-pg16-github-actions-selfhosted]"
Flaky tests (4)

Postgres 17

Postgres 14

Code coverage* (full report)

  • functions: 33.5% (8497 of 25347 functions)
  • lines: 49.3% (71492 of 145077 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
3d36dfe at 2025-01-27T21:45:10.446Z :recycle:

Please sign in to comment.