-
Notifications
You must be signed in to change notification settings - Fork 18
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Better logs/metrics when divergent vertex execution is detected (#830)
- Loading branch information
Showing
9 changed files
with
368 additions
and
175 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
176 changes: 176 additions & 0 deletions
176
core/src/main/java/com/radixdlt/consensus/DivergentVertexExecutionDetector.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,176 @@ | ||
| /* Copyright 2021 Radix Publishing Ltd incorporated in Jersey (Channel Islands). | ||
| * | ||
| * Licensed under the Radix License, Version 1.0 (the "License"); you may not use this | ||
| * file except in compliance with the License. You may obtain a copy of the License at: | ||
| * | ||
| * radixfoundation.org/licenses/LICENSE-v1 | ||
| * | ||
| * The Licensor hereby grants permission for the Canonical version of the Work to be | ||
| * published, distributed and used under or by reference to the Licensor’s trademark | ||
| * Radix ® and use of any unregistered trade names, logos or get-up. | ||
| * | ||
| * The Licensor provides the Work (and each Contributor provides its Contributions) on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, | ||
| * including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, | ||
| * MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. | ||
| * | ||
| * Whilst the Work is capable of being deployed, used and adopted (instantiated) to create | ||
| * a distributed ledger it is your responsibility to test and validate the code, together | ||
| * with all logic and performance of that code under all foreseeable scenarios. | ||
| * | ||
| * The Licensor does not make or purport to make and hereby excludes liability for all | ||
| * and any representation, warranty or undertaking in any form whatsoever, whether express | ||
| * or implied, to any entity or person, including any representation, warranty or | ||
| * undertaking, as to the functionality security use, value or other characteristics of | ||
| * any distributed ledger nor in respect the functioning or value of any tokens which may | ||
| * be created stored or transferred using the Work. The Licensor does not warrant that the | ||
| * Work or any use of the Work complies with any law or regulation in any territory where | ||
| * it may be implemented or used or that it will be appropriate for any specific purpose. | ||
| * | ||
| * Neither the licensor nor any current or former employees, officers, directors, partners, | ||
| * trustees, representatives, agents, advisors, contractors, or volunteers of the Licensor | ||
| * shall be liable for any direct or indirect, special, incidental, consequential or other | ||
| * losses of any kind, in tort, contract or otherwise (including but not limited to loss | ||
| * of revenue, income or profits, or loss of use or data, or loss of reputation, or loss | ||
| * of any economic or other opportunity of whatsoever nature or howsoever arising), arising | ||
| * out of or in connection with (without limitation of any use, misuse, of any ledger system | ||
| * or use made or its functionality or any performance or operation of any code or protocol | ||
| * caused by bugs or programming or logic errors or otherwise); | ||
| * | ||
| * A. any offer, purchase, holding, use, sale, exchange or transmission of any | ||
| * cryptographic keys, tokens or assets created, exchanged, stored or arising from any | ||
| * interaction with the Work; | ||
| * | ||
| * B. any failure in a transmission or loss of any token or assets keys or other digital | ||
| * artefacts due to errors in transmission; | ||
| * | ||
| * C. bugs, hacks, logic errors or faults in the Work or any communication; | ||
| * | ||
| * D. system software or apparatus including but not limited to losses caused by errors | ||
| * in holding or transmitting tokens by any third-party; | ||
| * | ||
| * E. breaches or failure of security including hacker attacks, loss or disclosure of | ||
| * password, loss of private key, unauthorised use or misuse of such passwords or keys; | ||
| * | ||
| * F. any losses including loss of anticipated savings or other benefits resulting from | ||
| * use of the Work or any changes to the Work (however implemented). | ||
| * | ||
| * You are solely responsible for; testing, validating and evaluation of all operation | ||
| * logic, functionality, security and appropriateness of using the Work for any commercial | ||
| * or non-commercial purpose and for any reproduction or redistribution by You of the | ||
| * Work. You assume all risks associated with Your use of the Work and the exercise of | ||
| * permissions under this License. | ||
| */ | ||
|
|
||
| package com.radixdlt.consensus; | ||
|
|
||
| import com.google.common.hash.HashCode; | ||
| import com.google.common.util.concurrent.RateLimiter; | ||
| import com.radixdlt.consensus.bft.BFTValidatorId; | ||
| import com.radixdlt.consensus.bft.BFTValidatorSet; | ||
| import com.radixdlt.consensus.bft.Round; | ||
| import com.radixdlt.monitoring.Metrics; | ||
| import java.math.BigDecimal; | ||
| import java.math.RoundingMode; | ||
| import java.util.*; | ||
| import java.util.stream.Collectors; | ||
| import org.apache.logging.log4j.LogManager; | ||
| import org.apache.logging.log4j.Logger; | ||
|
|
||
| /** | ||
| * A utility for detecting divergent vertex execution, i.e. a situation where two (or more) | ||
| * validators from the validator set have voted for the same vertexId, but their resultant ledger | ||
| * headers were different. | ||
| */ | ||
| @SuppressWarnings("UnstableApiUsage") | ||
| public final class DivergentVertexExecutionDetector { | ||
| private static final Logger log = LogManager.getLogger(); | ||
|
|
||
| private final RateLimiter logRatelimiter = RateLimiter.create(0.05); // At most one log every 20s | ||
|
|
||
| private final Metrics metrics; | ||
| private final BFTValidatorSet validatorSet; | ||
|
|
||
| // vertexId -> ledgerHeader -> validators who voted for (vertexId, ledgerHeader) | ||
| private final Map<HashCode, Map<LedgerHeader, Set<BFTValidatorId>>> votesByVertexId = | ||
| new HashMap<>(); | ||
|
|
||
| public DivergentVertexExecutionDetector(Metrics metrics, BFTValidatorSet validatorSet) { | ||
| this.metrics = metrics; | ||
| this.validatorSet = validatorSet; | ||
| } | ||
|
|
||
| /** | ||
| * Processes a received vote. No additional filtering is applied, the caller should ensure that | ||
| * only relevant votes are being processed (e.g. only votes from a single round). | ||
| */ | ||
| public void processVote(Vote vote) { | ||
| final var ledgerHeadersByVertexId = | ||
| votesByVertexId.computeIfAbsent( | ||
| vote.getVoteData().getProposed().getVertexId(), unused -> new HashMap<>()); | ||
| final var authorsByLedgerHeader = | ||
| ledgerHeadersByVertexId.computeIfAbsent( | ||
| vote.getVoteData().getProposed().getLedgerHeader(), unused -> new HashSet<>()); | ||
| authorsByLedgerHeader.add(vote.getAuthor()); | ||
| } | ||
|
|
||
| public void summarizeAfterRoundAndReset(Round round) { | ||
| // Divergent executions are the ones that have more than one resultant header | ||
| // for the same vertexId. | ||
| final StringBuilder logBuilder = new StringBuilder(); | ||
| votesByVertexId.entrySet().stream() | ||
| .filter(e -> e.getValue().size() > 1) | ||
| .forEach( | ||
| e -> { | ||
| final var vertexId = e.getKey(); | ||
| final var distinctResults = e.getValue(); | ||
|
|
||
| metrics | ||
| .bft() | ||
| .divergentVertexExecutions() | ||
| .label(new Metrics.Bft.DivergentVertexExecution(distinctResults.size())) | ||
| .inc(); | ||
|
|
||
| logBuilder.append( | ||
| String.format( | ||
| "In round %s validators have voted for vertex %s but they've computed %s" | ||
| + " distinct results:\n", | ||
| round, vertexId, distinctResults.size())); | ||
| final var totalStakeDec = new BigDecimal(validatorSet.getTotalPower().toBigInt()); | ||
| for (var result : distinctResults.entrySet()) { | ||
| final var stakeVoted = | ||
| result.getValue().stream() | ||
| .map(v -> new BigDecimal(validatorSet.getPower(v).toBigInt())) | ||
| .reduce(BigDecimal.ZERO, BigDecimal::add); | ||
| final var stakeVotedProportion = | ||
| stakeVoted.divide(totalStakeDec, 4, RoundingMode.HALF_UP); | ||
| // Let's list the actual validators if they represent less than 10% stake | ||
| final var validatorsDetails = | ||
| stakeVotedProportion.compareTo(BigDecimal.valueOf(0.1)) < 0 | ||
| ? " (" | ||
| + result.getValue().stream() | ||
| .map(BFTValidatorId::toString) | ||
| .collect(Collectors.joining(",")) | ||
| + ")" | ||
| : ""; | ||
| logBuilder.append( | ||
| String.format( | ||
| " * %s validator(s)%s representing %s stake computed %s\n", | ||
| result.getValue().size(), | ||
| validatorsDetails, | ||
| stakeVotedProportion, | ||
| result.getKey())); | ||
| } | ||
| }); | ||
|
|
||
| if (logBuilder.length() > 0 && logRatelimiter.tryAcquire()) { | ||
| logBuilder.append( | ||
| "This is informational only, this node is unaffected unless other error messages" | ||
| + " follow.\n"); | ||
| log.info(logBuilder.toString()); | ||
| } | ||
|
|
||
| // Reset this detector to its initial (empty) state | ||
| this.votesByVertexId.clear(); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.