Skip to content

CometBFT PeerState JSON serialization deadlock

Moderate severity GitHub Reviewed Published Jul 3, 2023 in cometbft/cometbft • Updated Nov 5, 2023

Package

gomod github.com/cometbft/cometbft (Go)

Affected versions

= 0.34.28
= 0.37.1

Patched versions

0.34.29
0.37.2

Description

Impact

An internal modification to the way struct PeerState is serialized to JSON introduced a deadlock when new function MarshallJSON is called. This function can be called from two places:

  1. Via logs
    • Setting the consensus logging module to "debug" level (should not happen in production), and
    • Setting the log output format to JSON
  2. Via RPC dump_consensus_state

Case 1 above, which should not be hit in production, will eventually hit the deadlock in most goroutines, effectively halting the node.

In case 2, only the data structures related to the first peer will be deadlocked, together with the thread(s) dealing with the RPC request(s). This means that only one of the channels of communication to the node's peers will be blocked. Eventually the peer will timeout and excluded from the list (typically after 2 minutes). The goroutines involved in the deadlock will not be garbage collected, but they will not interfere with the system after the peer is excluded.

The theoretical worst case for case 2, is a network with only two validator nodes. In this case, each of the nodes only has one PeerState struct. If dump_consensus_state is called in either node (or both), the chain will halt until the peer connections time out, after which the nodes will reconnect (with different PeerState structs) and the chain will progress again. Then, the same process can be repeated.

As the number of nodes in a network increases, and thus, the number of peer struct each node maintains, the possibility of reproducing the perturbation visible with 2 nodes decreases. Only the first PeerState struct will deadlock, and not the others (RPC dump_consensus_state accesses them in a for loop, so the deadlock at the first iteration causes the rest of the iterations of that "for" loop to never be reached).

This regression was introduced in versions v0.34.28 and v0.37.1, and will be fixed in v0.34.29 and v0.37.2.

Patches

The PR containing the fix is here, and the corresponding issue is here

Workarounds

For case 1 (hitting the deadlock via logs)

  • either don't set the log output to "json", leave at "plain",
  • or don't set the consensus logging module to "debug", leave it at "info" or higher.

For case 2 (hitting the deadlock via RPC dump_consensus_state)

  • do not expose dump_consensus_state RPC endpoint to the public internet (e.g., via rules in your nginx setup)

References

  • Issue that introduced the deadlock
  • Issue reporting the bug via logs

References

@sergio-mena sergio-mena published to cometbft/cometbft Jul 3, 2023
Published by the National Vulnerability Database Jul 3, 2023
Published to the GitHub Advisory Database Jul 5, 2023
Reviewed Jul 5, 2023
Last updated Nov 5, 2023

Severity

Moderate

CVSS overall score

This score calculates overall vulnerability severity from 0 to 10 and is based on the Common Vulnerability Scoring System (CVSS).
/ 10

CVSS v3 base metrics

Attack vector
Network
Attack complexity
Low
Privileges required
None
User interaction
None
Scope
Unchanged
Confidentiality
None
Integrity
None
Availability
Low

CVSS v3 base metrics

Attack vector: More severe the more the remote (logically and physically) an attacker can be in order to exploit the vulnerability.
Attack complexity: More severe for the least complex attacks.
Privileges required: More severe if no privileges are required.
User interaction: More severe when no user interaction is required.
Scope: More severe when a scope change occurs, e.g. one vulnerable component impacts resources in components beyond its security scope.
Confidentiality: More severe when loss of data confidentiality is highest, measuring the level of data access available to an unauthorized user.
Integrity: More severe when loss of data integrity is the highest, measuring the consequence of data modification possible by an unauthorized user.
Availability: More severe when the loss of impacted component availability is highest.
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

EPSS score

0.072%
(32nd percentile)

CVE ID

CVE-2023-34450

GHSA ID

GHSA-mvj3-qrqh-cjvr

Source code

Credits

Loading Checking history
See something to contribute? Suggest improvements for this vulnerability.