diff --git a/CHANGELOG.md b/CHANGELOG.md index f966114ed92..2ab5425cd19 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,16 +4,21 @@ ### Breaking Changes - The `trace-filter` method in JSON-RPC API now has a default block range limit of 1000, adjustable with `--rpc-max-trace-filter-range` [#6446](https://github.com/hyperledger/besu/pull/6446) +- Requesting the Ethereum Node Record (ENR) to acquire the fork id from bonded peers is now enabled by default, so the following change has been made [#5628](https://github.com/hyperledger/besu/pull/5628): + - `--Xfilter-on-enr-fork-id` has been removed. To disable the feature use `--filter-on-enr-fork-id=false`. ### Deprecations ### Additions and Improvements - Add `OperationTracer.tracePrepareTransaction`, where the sender account has not yet been altered[#6453](https://github.com/hyperledger/besu/pull/6453) - Improve the high spec flag by limiting it to a few column families [#6354](https://github.com/hyperledger/besu/pull/6354) - +- Log blob count when importing a block via Engine API [#6466](https://github.com/hyperledger/besu/pull/6466) +- Introduce `--Xbonsai-limit-trie-logs-enabled` experimental feature which by default will only retain the latest 512 trie logs, saving about 3GB per week in database growth [#5390](https://github.com/hyperledger/besu/issues/5390) +- Introduce `besu storage x-trie-log prune` experimental offline subcommand which will prune all redundant trie logs except the latest 512 [#6303](https://github.com/hyperledger/besu/pull/6303) ### Bug fixes - Fix the way an advertised host configured with `--p2p-host` is treated when communicating with the originator of a PING packet [#6225](https://github.com/hyperledger/besu/pull/6225) +- Fix `poa-block-txs-selection-max-time` option that was inadvertently reset to its default after being configured [#6444](https://github.com/hyperledger/besu/pull/6444) ### Download Links @@ -23,9 +28,7 @@ - New `EXECUTION_HALTED` error returned if there is an error executing or simulating a transaction, with the reason for execution being halted. Replaces the generic `INTERNAL_ERROR` return code in certain cases which some applications may be checking for [#6343](https://github.com/hyperledger/besu/pull/6343) - The Besu Docker images with `openjdk-latest` tags since 23.10.3 were incorrectly using UID 1001 instead of 1000 for the container's `besu` user. The user now uses 1000 again. Containers created from or migrated to images using UID 1001 will need to chown their persistent database files to UID 1000 [#6360](https://github.com/hyperledger/besu/pull/6360) - The deprecated `--privacy-onchain-groups-enabled` option has now been removed. Use the `--privacy-flexible-groups-enabled` option instead. [#6411](https://github.com/hyperledger/besu/pull/6411) -- Requesting the Ethereum Node Record (ENR) to acquire the fork id from bonded peers is now enabled by default, so the following change has been made [#5628](https://github.com/hyperledger/besu/pull/5628): - - `--Xfilter-on-enr-fork-id` has been removed. To disable the feature use `--filter-on-enr-fork-id=false`. -- The time that can be spent selecting transactions during block creation is not capped at 5 seconds for PoS and PoW networks, and for PoA networks, at 75% of the block period specified in the genesis, this to prevent possible DoS in case a single transaction is taking too long to execute, and to have a stable block production rate, but it could be a breaking change if an existing network used to have transactions that takes more time to executed that the newly introduced limit, if it is mandatory for these network to keep processing these long processing transaction, then the default value of `block-txs-selection-max-time` or `poa-block-txs-selection-max-time` needs to be tuned accordingly. +- The time that can be spent selecting transactions during block creation is not capped at 5 seconds for PoS and PoW networks, and for PoA networks, at 75% of the block period specified in the genesis, this to prevent possible DoS in case a single transaction is taking too long to execute, and to have a stable block production rate, but it could be a breaking change if an existing network used to have transactions that takes more time to executed that the newly introduced limit, if it is mandatory for these network to keep processing these long processing transaction, then the default value of `block-txs-selection-max-time` or `poa-block-txs-selection-max-time` needs to be tuned accordingly. [#6423](https://github.com/hyperledger/besu/pull/6423) ### Deprecations @@ -39,8 +42,6 @@ - Upgrade Mockito [#6397](https://github.com/hyperledger/besu/pull/6397) - Upgrade `tech.pegasys.discovery:discovery` [#6414](https://github.com/hyperledger/besu/pull/6414) - Options to tune the max allowed time that can be spent selecting transactions during block creation are now stable [#6423](https://github.com/hyperledger/besu/pull/6423) -- Introduce `--Xbonsai-limit-trie-logs-enabled` experimental feature which by default will only retain the latest 512 trie logs, saving about 3GB per week in database growth [#5390](https://github.com/hyperledger/besu/issues/5390) -- Introduce `besu storage x-trie-log prune` experimental offline subcommand which will prune all redundant trie logs except the latest 512 [#6303](https://github.com/hyperledger/besu/pull/6303) ### Bug fixes - INTERNAL_ERROR from `eth_estimateGas` JSON/RPC calls [#6344](https://github.com/hyperledger/besu/issues/6344) @@ -48,8 +49,12 @@ - Fluent EVM API definition for Tangerine Whistle had incorrect code size validation configured [#6382](https://github.com/hyperledger/besu/pull/6382) - Correct mining beneficiary for Clique networks in TraceServiceImpl [#6390](https://github.com/hyperledger/besu/pull/6390) - Fix to gas limit delta calculations used in block production. Besu should now increment or decrement the block gas limit towards its target correctly (thanks @arbora) #6425 +- Ensure Backward Sync waits for initial sync before starting a session [#6455](https://github.com/hyperledger/besu/issues/6455) +- Silence the noisy DNS query errors [#6458](https://github.com/hyperledger/besu/issues/6458) ### Download Links +https://hyperledger.jfrog.io/artifactory/besu-binaries/besu/24.1.1/besu-24.1.1.zip / sha256 b6b64f939e0bb4937ce90fc647e0a7073ce3e359c10352b502059955070a60c6 +https://hyperledger.jfrog.io/artifactory/besu-binaries/besu/24.1.1/besu-24.1.1.tar.gz / sha256 cfcae04c30769bf338b0740ac65870f9346d3469931bb46cdba3b2f65d311e7a ## 24.1.0 @@ -72,8 +77,8 @@ - mitigation for trielog failure [#6315]((https://github.com/hyperledger/besu/pull/6315) ### Download Links -https://hyperledger.jfrog.io/artifactory/besu-binaries/besu/24.1.0/besu-24.1.0.zip / sha256 TBA -https://hyperledger.jfrog.io/artifactory/besu-binaries/besu/24.1.0/besu-24.1.0.tar.gz / sha256 TBA +https://hyperledger.jfrog.io/artifactory/besu-binaries/besu/24.1.0/besu-24.1.0.zip / sha256 d36c8aeef70f0a516d4c26d3bc696c3e2a671e515c9e6e9475a31fe759e39f64 +https://hyperledger.jfrog.io/artifactory/besu-binaries/besu/24.1.0/besu-24.1.0.tar.gz / sha256 602b04c0729a7b17361d1f0b39f4ce6a2ebe47932165add666560fe594d9ca99 ## 23.10.3-hotfix diff --git a/besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java b/besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java index d9a9e0c6f09..68c3a6a5abc 100644 --- a/besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java +++ b/besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java @@ -116,7 +116,6 @@ import org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketConfiguration; import org.hyperledger.besu.ethereum.api.query.BlockchainQueries; import org.hyperledger.besu.ethereum.chain.Blockchain; -import org.hyperledger.besu.ethereum.core.ImmutableMiningParameters; import org.hyperledger.besu.ethereum.core.MiningParameters; import org.hyperledger.besu.ethereum.core.PrivacyParameters; import org.hyperledger.besu.ethereum.eth.sync.SyncMode; @@ -215,6 +214,7 @@ import java.util.Map; import java.util.Objects; import java.util.Optional; +import java.util.OptionalInt; import java.util.Set; import java.util.TreeMap; import java.util.function.Function; @@ -2401,32 +2401,28 @@ private TransactionPoolConfiguration buildTransactionPoolConfiguration() { private MiningParameters getMiningParameters() { if (miningParameters == null) { - final var miningParametersBuilder = - ImmutableMiningParameters.builder().from(miningOptions.toDomainObject()); - final var actualGenesisOptions = getActualGenesisConfigOptions(); - if (actualGenesisOptions.isPoa()) { - miningParametersBuilder.genesisBlockPeriodSeconds( - getGenesisBlockPeriodSeconds(actualGenesisOptions)); - } - miningParameters = miningParametersBuilder.build(); + miningOptions.setGenesisBlockPeriodSeconds( + getGenesisBlockPeriodSeconds(getActualGenesisConfigOptions())); + miningParameters = miningOptions.toDomainObject(); } return miningParameters; } - private int getGenesisBlockPeriodSeconds(final GenesisConfigOptions genesisConfigOptions) { + private OptionalInt getGenesisBlockPeriodSeconds( + final GenesisConfigOptions genesisConfigOptions) { if (genesisConfigOptions.isClique()) { - return genesisConfigOptions.getCliqueConfigOptions().getBlockPeriodSeconds(); + return OptionalInt.of(genesisConfigOptions.getCliqueConfigOptions().getBlockPeriodSeconds()); } if (genesisConfigOptions.isIbft2()) { - return genesisConfigOptions.getBftConfigOptions().getBlockPeriodSeconds(); + return OptionalInt.of(genesisConfigOptions.getBftConfigOptions().getBlockPeriodSeconds()); } if (genesisConfigOptions.isQbft()) { - return genesisConfigOptions.getQbftConfigOptions().getBlockPeriodSeconds(); + return OptionalInt.of(genesisConfigOptions.getQbftConfigOptions().getBlockPeriodSeconds()); } - throw new IllegalArgumentException("Should only be called for a PoA network"); + return OptionalInt.empty(); } private boolean isPruningEnabled() { @@ -2878,7 +2874,12 @@ private Optional getEcCurveFromGenesisFile() { return genesisConfigOptions.getEcCurve(); } - private GenesisConfigOptions getActualGenesisConfigOptions() { + /** + * Return the genesis config options after applying any specified config overrides + * + * @return the genesis config options after applying any specified config overrides + */ + protected GenesisConfigOptions getActualGenesisConfigOptions() { return Optional.ofNullable(genesisConfigOptions) .orElseGet( () -> diff --git a/besu/src/main/java/org/hyperledger/besu/cli/options/MiningOptions.java b/besu/src/main/java/org/hyperledger/besu/cli/options/MiningOptions.java index 55197b8e826..2e5a239f16d 100644 --- a/besu/src/main/java/org/hyperledger/besu/cli/options/MiningOptions.java +++ b/besu/src/main/java/org/hyperledger/besu/cli/options/MiningOptions.java @@ -40,6 +40,7 @@ import org.hyperledger.besu.util.number.PositiveNumber; import java.util.List; +import java.util.OptionalInt; import org.apache.tuweni.bytes.Bytes; import org.slf4j.Logger; @@ -188,6 +189,8 @@ static class Unstable { DEFAULT_POS_BLOCK_CREATION_REPETITION_MIN_DURATION; } + private OptionalInt maybeGenesisBlockPeriodSeconds; + private MiningOptions() {} /** @@ -199,6 +202,16 @@ public static MiningOptions create() { return new MiningOptions(); } + /** + * Set the optional genesis block period per seconds + * + * @param genesisBlockPeriodSeconds if the network is PoA then the block period in seconds + * specified in the genesis file, otherwise empty. + */ + public void setGenesisBlockPeriodSeconds(final OptionalInt genesisBlockPeriodSeconds) { + maybeGenesisBlockPeriodSeconds = genesisBlockPeriodSeconds; + } + /** * Validate that there are no inconsistencies in the specified options. For example that the * options are valid for the selected implementation. @@ -285,6 +298,7 @@ public void validate( static MiningOptions fromConfig(final MiningParameters miningParameters) { final MiningOptions miningOptions = MiningOptions.create(); + miningOptions.setGenesisBlockPeriodSeconds(miningParameters.getGenesisBlockPeriodSeconds()); miningOptions.isMiningEnabled = miningParameters.isMiningEnabled(); miningOptions.iStratumMiningEnabled = miningParameters.isStratumMiningEnabled(); miningOptions.stratumNetworkInterface = miningParameters.getStratumNetworkInterface(); @@ -319,6 +333,11 @@ static MiningOptions fromConfig(final MiningParameters miningParameters) { @Override public MiningParameters toDomainObject() { + if (maybeGenesisBlockPeriodSeconds == null) { + throw new IllegalStateException( + "genesisBlockPeriodSeconds must be set before using this object"); + } + final var updatableInitValuesBuilder = MutableInitValues.builder() .isMiningEnabled(isMiningEnabled) @@ -334,27 +353,26 @@ public MiningParameters toDomainObject() { updatableInitValuesBuilder.coinbase(coinbase); } - final var miningParametersBuilder = - ImmutableMiningParameters.builder() - .mutableInitValues(updatableInitValuesBuilder.build()) - .isStratumMiningEnabled(iStratumMiningEnabled) - .stratumNetworkInterface(stratumNetworkInterface) - .stratumPort(stratumPort) - .nonPoaBlockTxsSelectionMaxTime(nonPoaBlockTxsSelectionMaxTime) - .poaBlockTxsSelectionMaxTime(poaBlockTxsSelectionMaxTime) - .unstable( - ImmutableMiningParameters.Unstable.builder() - .remoteSealersLimit(unstableOptions.remoteSealersLimit) - .remoteSealersTimeToLive(unstableOptions.remoteSealersTimeToLive) - .powJobTimeToLive(unstableOptions.powJobTimeToLive) - .maxOmmerDepth(unstableOptions.maxOmmersDepth) - .stratumExtranonce(unstableOptions.stratumExtranonce) - .posBlockCreationMaxTime(unstableOptions.posBlockCreationMaxTime) - .posBlockCreationRepetitionMinDuration( - unstableOptions.posBlockCreationRepetitionMinDuration) - .build()); - - return miningParametersBuilder.build(); + return ImmutableMiningParameters.builder() + .genesisBlockPeriodSeconds(maybeGenesisBlockPeriodSeconds) + .mutableInitValues(updatableInitValuesBuilder.build()) + .isStratumMiningEnabled(iStratumMiningEnabled) + .stratumNetworkInterface(stratumNetworkInterface) + .stratumPort(stratumPort) + .nonPoaBlockTxsSelectionMaxTime(nonPoaBlockTxsSelectionMaxTime) + .poaBlockTxsSelectionMaxTime(poaBlockTxsSelectionMaxTime) + .unstable( + ImmutableMiningParameters.Unstable.builder() + .remoteSealersLimit(unstableOptions.remoteSealersLimit) + .remoteSealersTimeToLive(unstableOptions.remoteSealersTimeToLive) + .powJobTimeToLive(unstableOptions.powJobTimeToLive) + .maxOmmerDepth(unstableOptions.maxOmmersDepth) + .stratumExtranonce(unstableOptions.stratumExtranonce) + .posBlockCreationMaxTime(unstableOptions.posBlockCreationMaxTime) + .posBlockCreationRepetitionMinDuration( + unstableOptions.posBlockCreationRepetitionMinDuration) + .build()) + .build(); } @Override diff --git a/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java index 88d876cce2e..22efd97c86d 100644 --- a/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java +++ b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java @@ -16,7 +16,9 @@ package org.hyperledger.besu.cli.subcommands.storage; import static com.google.common.base.Preconditions.checkArgument; +import static org.hyperledger.besu.cli.options.stable.DataStorageOptions.BONSAI_STORAGE_FORMAT_MAX_LAYERS_TO_LOAD; import static org.hyperledger.besu.controller.BesuController.DATABASE_PATH; +import static org.hyperledger.besu.ethereum.worldstate.DataStorageConfiguration.Unstable.DEFAULT_BONSAI_TRIE_LOG_PRUNING_WINDOW_SIZE; import org.hyperledger.besu.cli.options.stable.DataStorageOptions; import org.hyperledger.besu.datatypes.Hash; @@ -74,11 +76,16 @@ void prune( final long lastBlockNumberToRetainTrieLogsFor = chainHeight - layersToRetain + 1; - if (!validPruneRequirements(blockchain, chainHeight, lastBlockNumberToRetainTrieLogsFor)) { + if (!validatePruneRequirements( + blockchain, + chainHeight, + lastBlockNumberToRetainTrieLogsFor, + rootWorldStateStorage, + layersToRetain)) { return; } - final long numberOfBatches = calculateNumberofBatches(layersToRetain); + final long numberOfBatches = calculateNumberOfBatches(layersToRetain); processTrieLogBatches( rootWorldStateStorage, @@ -88,11 +95,23 @@ void prune( numberOfBatches, batchFileNameBase); - if (rootWorldStateStorage.streamTrieLogKeys(layersToRetain).count() == layersToRetain) { - deleteFiles(batchFileNameBase, numberOfBatches); - LOG.info("Prune ran successfully. Enjoy some disk space back! \uD83D\uDE80"); + // Should only be layersToRetain left but loading extra just in case of an unforeseen bug + final long countAfterPrune = + rootWorldStateStorage + .streamTrieLogKeys(layersToRetain + DEFAULT_BONSAI_TRIE_LOG_PRUNING_WINDOW_SIZE) + .count(); + if (countAfterPrune == layersToRetain) { + if (deleteFiles(batchFileNameBase, numberOfBatches)) { + LOG.info("Prune ran successfully. Enjoy some disk space back! \uD83D\uDE80"); + } else { + throw new IllegalStateException( + "There was an error deleting the trie log backup files. Please ensure besu is working before deleting them manually."); + } } else { - LOG.error("Prune failed. Re-run the subcommand to load the trie logs from file."); + throw new IllegalStateException( + String.format( + "Remaining trie logs (%d) did not match %s (%d). Trie logs backup files have not been deleted, it is safe to rerun the subcommand.", + countAfterPrune, BONSAI_STORAGE_FORMAT_MAX_LAYERS_TO_LOAD, layersToRetain)); } } @@ -151,15 +170,21 @@ private void restoreTrieLogBatches( } } - private void deleteFiles(final String batchFileNameBase, final long numberOfBatches) { + private boolean deleteFiles(final String batchFileNameBase, final long numberOfBatches) { LOG.info("Deleting files..."); - for (long batchNumber = 1; batchNumber <= numberOfBatches; batchNumber++) { - File file = new File(batchFileNameBase + "-" + batchNumber); - if (file.exists()) { - file.delete(); + try { + for (long batchNumber = 1; batchNumber <= numberOfBatches; batchNumber++) { + File file = new File(batchFileNameBase + "-" + batchNumber); + if (file.exists()) { + file.delete(); + } } + return true; + } catch (Exception e) { + LOG.error("Error deleting files", e); + return false; } } @@ -177,14 +202,17 @@ private List getTrieLogKeysForBlocks( return trieLogKeys; } - private long calculateNumberofBatches(final long layersToRetain) { + private long calculateNumberOfBatches(final long layersToRetain) { return layersToRetain / BATCH_SIZE + ((layersToRetain % BATCH_SIZE == 0) ? 0 : 1); } - private boolean validPruneRequirements( + private boolean validatePruneRequirements( final MutableBlockchain blockchain, final long chainHeight, - final long lastBlockNumberToRetainTrieLogsFor) { + final long lastBlockNumberToRetainTrieLogsFor, + final BonsaiWorldStateKeyValueStorage rootWorldStateStorage, + final long layersToRetain) { + if (lastBlockNumberToRetainTrieLogsFor < 0) { throw new IllegalArgumentException( "Trying to retain more trie logs than chain length (" @@ -192,6 +220,19 @@ private boolean validPruneRequirements( + "), skipping pruning"); } + // Need to ensure we're loading at least layersToRetain if they exist + // plus extra threshold to account forks and orphans + final long clampedCountBeforePruning = + rootWorldStateStorage + .streamTrieLogKeys(layersToRetain + DEFAULT_BONSAI_TRIE_LOG_PRUNING_WINDOW_SIZE) + .count(); + if (clampedCountBeforePruning < layersToRetain) { + throw new IllegalArgumentException( + String.format( + "Trie log count (%d) is less than retention limit (%d), skipping pruning", + clampedCountBeforePruning, layersToRetain)); + } + final Optional finalizedBlockHash = blockchain.getFinalized(); if (finalizedBlockHash.isEmpty()) { @@ -250,7 +291,7 @@ void validatePruneConfiguration(final DataStorageConfiguration config) { config.getBonsaiMaxLayersToLoad() >= DataStorageConfiguration.Unstable.MINIMUM_BONSAI_TRIE_LOG_RETENTION_LIMIT, String.format( - DataStorageOptions.BONSAI_STORAGE_FORMAT_MAX_LAYERS_TO_LOAD + " minimum value is %d", + BONSAI_STORAGE_FORMAT_MAX_LAYERS_TO_LOAD + " minimum value is %d", DataStorageConfiguration.Unstable.MINIMUM_BONSAI_TRIE_LOG_RETENTION_LIMIT)); checkArgument( config.getUnstable().getBonsaiTrieLogPruningWindowSize() > 0, @@ -264,7 +305,7 @@ void validatePruneConfiguration(final DataStorageConfiguration config) { String.format( DataStorageOptions.Unstable.BONSAI_TRIE_LOG_PRUNING_WINDOW_SIZE + "=%d must be greater than " - + DataStorageOptions.BONSAI_STORAGE_FORMAT_MAX_LAYERS_TO_LOAD + + BONSAI_STORAGE_FORMAT_MAX_LAYERS_TO_LOAD + "=%d", config.getUnstable().getBonsaiTrieLogPruningWindowSize(), config.getBonsaiMaxLayersToLoad())); diff --git a/besu/src/main/java/org/hyperledger/besu/cli/util/TomlConfigurationDefaultProvider.java b/besu/src/main/java/org/hyperledger/besu/cli/util/TomlConfigurationDefaultProvider.java index 4928d3f7ff6..a346e260d13 100644 --- a/besu/src/main/java/org/hyperledger/besu/cli/util/TomlConfigurationDefaultProvider.java +++ b/besu/src/main/java/org/hyperledger/besu/cli/util/TomlConfigurationDefaultProvider.java @@ -17,6 +17,7 @@ import org.hyperledger.besu.datatypes.Wei; import org.hyperledger.besu.util.number.Fraction; import org.hyperledger.besu.util.number.Percentage; +import org.hyperledger.besu.util.number.PositiveNumber; import java.io.File; import java.io.FileInputStream; @@ -129,7 +130,8 @@ private boolean isNumericType(final Class type) { || type.equals(Float.class) || type.equals(float.class) || type.equals(Percentage.class) - || type.equals(Fraction.class); + || type.equals(Fraction.class) + || type.equals(PositiveNumber.class); } private String getEntryAsString(final OptionSpec spec) { diff --git a/besu/src/test/java/org/hyperledger/besu/cli/CommandTestAbstract.java b/besu/src/test/java/org/hyperledger/besu/cli/CommandTestAbstract.java index 2c5f68be8c5..1609758b966 100644 --- a/besu/src/test/java/org/hyperledger/besu/cli/CommandTestAbstract.java +++ b/besu/src/test/java/org/hyperledger/besu/cli/CommandTestAbstract.java @@ -44,6 +44,7 @@ import org.hyperledger.besu.cli.options.unstable.NetworkingOptions; import org.hyperledger.besu.cli.options.unstable.SynchronizerOptions; import org.hyperledger.besu.components.BesuComponent; +import org.hyperledger.besu.config.GenesisConfigOptions; import org.hyperledger.besu.consensus.qbft.pki.PkiBlockCreationConfiguration; import org.hyperledger.besu.consensus.qbft.pki.PkiBlockCreationConfigurationProvider; import org.hyperledger.besu.controller.BesuController; @@ -133,20 +134,37 @@ @ExtendWith(MockitoExtension.class) public abstract class CommandTestAbstract { private static final Logger TEST_LOGGER = LoggerFactory.getLogger(CommandTestAbstract.class); + + protected static final int POA_BLOCK_PERIOD_SECONDS = 5; protected static final JsonObject VALID_GENESIS_QBFT_POST_LONDON = (new JsonObject()) .put( "config", new JsonObject() .put("londonBlock", 0) - .put("qbft", new JsonObject().put("blockperiodseconds", 5))); + .put( + "qbft", + new JsonObject().put("blockperiodseconds", POA_BLOCK_PERIOD_SECONDS))); protected static final JsonObject VALID_GENESIS_IBFT2_POST_LONDON = (new JsonObject()) .put( "config", new JsonObject() .put("londonBlock", 0) - .put("ibft2", new JsonObject().put("blockperiodseconds", 5))); + .put( + "ibft2", + new JsonObject().put("blockperiodseconds", POA_BLOCK_PERIOD_SECONDS))); + + protected static final JsonObject VALID_GENESIS_CLIQUE_POST_LONDON = + (new JsonObject()) + .put( + "config", + new JsonObject() + .put("londonBlock", 0) + .put( + "clique", + new JsonObject().put("blockperiodseconds", POA_BLOCK_PERIOD_SECONDS))); + protected final PrintStream originalOut = System.out; protected final PrintStream originalErr = System.err; protected final ByteArrayOutputStream commandOutput = new ByteArrayOutputStream(); @@ -561,6 +579,11 @@ protected Vertx createVertx(final VertxOptions vertxOptions) { return vertx; } + @Override + public GenesisConfigOptions getActualGenesisConfigOptions() { + return super.getActualGenesisConfigOptions(); + } + public CommandSpec getSpec() { return spec; } diff --git a/besu/src/test/java/org/hyperledger/besu/cli/options/AbstractCLIOptionsTest.java b/besu/src/test/java/org/hyperledger/besu/cli/options/AbstractCLIOptionsTest.java index 92b190d74bf..21d8baf9ee9 100644 --- a/besu/src/test/java/org/hyperledger/besu/cli/options/AbstractCLIOptionsTest.java +++ b/besu/src/test/java/org/hyperledger/besu/cli/options/AbstractCLIOptionsTest.java @@ -67,7 +67,10 @@ private void getCLIOptions(final D domainObject) { final TestBesuCommand cmd = parseCommand(cliOptions); final T optionsFromCommand = getOptionsFromBesuCommand(cmd); - assertThat(optionsFromCommand).usingRecursiveComparison().isEqualTo(options); + assertThat(optionsFromCommand) + .usingRecursiveComparison() + .ignoringFields(getNonOptionFields()) + .isEqualTo(options); assertThat(commandOutput.toString(UTF_8)).isEmpty(); assertThat(commandErrorOutput.toString(UTF_8)).isEmpty(); @@ -82,10 +85,10 @@ public void defaultValues() { final T optionsFromCommand = getOptionsFromBesuCommand(cmd); // Check default values supplied by CLI match expected default values - final String[] fieldsToIgnore = getFieldsWithComputedDefaults().toArray(new String[0]); assertThat(optionsFromCommand) .usingRecursiveComparison() - .ignoringFields(fieldsToIgnore) + .ignoringFields(getFieldsWithComputedDefaults()) + .ignoringFields(getNonOptionFields()) .isEqualTo(defaultOptions); } @@ -93,8 +96,12 @@ public void defaultValues() { protected abstract D createCustomizedDomainObject(); - protected List getFieldsWithComputedDefaults() { - return Collections.emptyList(); + protected String[] getFieldsWithComputedDefaults() { + return new String[0]; + } + + protected String[] getNonOptionFields() { + return new String[0]; } protected List getFieldsToIgnore() { diff --git a/besu/src/test/java/org/hyperledger/besu/cli/options/MiningOptionsTest.java b/besu/src/test/java/org/hyperledger/besu/cli/options/MiningOptionsTest.java index 4b1fdddb53a..74b696cdf9a 100644 --- a/besu/src/test/java/org/hyperledger/besu/cli/options/MiningOptionsTest.java +++ b/besu/src/test/java/org/hyperledger/besu/cli/options/MiningOptionsTest.java @@ -32,7 +32,9 @@ import java.io.IOException; import java.nio.file.Path; +import java.time.Duration; import java.util.Optional; +import java.util.OptionalInt; import org.apache.tuweni.bytes.Bytes; import org.junit.jupiter.api.Test; @@ -361,13 +363,16 @@ public void poaBlockTxsSelectionMaxTimeOption() throws IOException { @Test public void poaBlockTxsSelectionMaxTimeOptionOver100Percent() throws IOException { - final Path genesisFileIBFT2 = createFakeGenesisFile(VALID_GENESIS_IBFT2_POST_LONDON); + final Path genesisFileClique = createFakeGenesisFile(VALID_GENESIS_CLIQUE_POST_LONDON); internalTestSuccess( - miningParams -> - assertThat(miningParams.getPoaBlockTxsSelectionMaxTime()) - .isEqualTo(PositiveNumber.fromInt(200)), + miningParams -> { + assertThat(miningParams.getPoaBlockTxsSelectionMaxTime()) + .isEqualTo(PositiveNumber.fromInt(200)); + assertThat(miningParams.getBlockTxsSelectionMaxTime()) + .isEqualTo(Duration.ofSeconds(POA_BLOCK_PERIOD_SECONDS * 2).toMillis()); + }, "--genesis-file", - genesisFileIBFT2.toString(), + genesisFileClique.toString(), "--poa-block-txs-selection-max-time", "200"); } @@ -407,6 +412,16 @@ protected MiningOptions optionsFromDomainObject(final MiningParameters domainObj @Override protected MiningOptions getOptionsFromBesuCommand(final TestBesuCommand besuCommand) { - return besuCommand.getMiningOptions(); + final var miningOptions = besuCommand.getMiningOptions(); + miningOptions.setGenesisBlockPeriodSeconds( + besuCommand.getActualGenesisConfigOptions().isPoa() + ? OptionalInt.of(POA_BLOCK_PERIOD_SECONDS) + : OptionalInt.empty()); + return miningOptions; + } + + @Override + protected String[] getNonOptionFields() { + return new String[] {"maybeGenesisBlockPeriodSeconds"}; } } diff --git a/besu/src/test/java/org/hyperledger/besu/cli/options/SynchronizerOptionsTest.java b/besu/src/test/java/org/hyperledger/besu/cli/options/SynchronizerOptionsTest.java index 3b1a9e6839a..9585fadf0d8 100644 --- a/besu/src/test/java/org/hyperledger/besu/cli/options/SynchronizerOptionsTest.java +++ b/besu/src/test/java/org/hyperledger/besu/cli/options/SynchronizerOptionsTest.java @@ -82,8 +82,8 @@ protected SynchronizerConfiguration.Builder createCustomizedDomainObject() { } @Override - protected List getFieldsWithComputedDefaults() { - return Arrays.asList("maxTrailingPeers", "computationParallelism"); + protected String[] getFieldsWithComputedDefaults() { + return new String[] {"maxTrailingPeers", "computationParallelism"}; } @Override diff --git a/besu/src/test/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelperTest.java b/besu/src/test/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelperTest.java index ca856ba0fad..22a7c37523d 100644 --- a/besu/src/test/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelperTest.java +++ b/besu/src/test/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelperTest.java @@ -18,8 +18,10 @@ import static java.util.Collections.singletonList; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.hyperledger.besu.ethereum.worldstate.DataStorageConfiguration.Unstable.DEFAULT_BONSAI_TRIE_LOG_PRUNING_WINDOW_SIZE; import static org.hyperledger.besu.ethereum.worldstate.DataStorageFormat.BONSAI; import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.spy; import static org.mockito.Mockito.when; import org.hyperledger.besu.datatypes.Hash; @@ -44,6 +46,7 @@ import java.util.Map; import java.util.Optional; import java.util.stream.Collectors; +import java.util.stream.Stream; import org.apache.tuweni.bytes.Bytes; import org.junit.jupiter.api.BeforeEach; @@ -58,7 +61,7 @@ class TrieLogHelperTest { private static final StorageProvider storageProvider = new InMemoryKeyValueStorageProvider(); private static BonsaiWorldStateKeyValueStorage inMemoryWorldState; - private TrieLogHelper trieLogHelper; + private TrieLogHelper nonValidatingTrieLogHelper; private static class NonValidatingTrieLogHelper extends TrieLogHelper { @Override @@ -106,7 +109,7 @@ public void setup() throws IOException { .put(blockHeader5.getHash().toArrayUnsafe(), createTrieLog(blockHeader5)); updater.getTrieLogStorageTransaction().commit(); - trieLogHelper = new NonValidatingTrieLogHelper(); + nonValidatingTrieLogHelper = new NonValidatingTrieLogHelper(); } private static byte[] createTrieLog(final BlockHeader blockHeader) { @@ -150,7 +153,8 @@ public void prune(final @TempDir Path dataDir) throws IOException { assertThat(inMemoryWorldState.getTrieLog(blockHeader3.getHash()).get()) .isEqualTo(createTrieLog(blockHeader3)); - trieLogHelper.prune(dataStorageConfiguration, inMemoryWorldState, blockchain, dataDir); + nonValidatingTrieLogHelper.prune( + dataStorageConfiguration, inMemoryWorldState, blockchain, dataDir); // assert pruned trie logs are not in the DB assertThat(inMemoryWorldState.getTrieLog(blockHeader1.getHash())).isEqualTo(Optional.empty()); @@ -182,7 +186,7 @@ public void cannotPruneIfNoFinalizedIsFound() { assertThatThrownBy( () -> - trieLogHelper.prune( + nonValidatingTrieLogHelper.prune( dataStorageConfiguration, inMemoryWorldState, blockchain, Path.of(""))) .isInstanceOf(RuntimeException.class) .hasMessage("No finalized block present, can't safely run trie log prune"); @@ -204,7 +208,7 @@ public void cannotPruneIfUserRetainsMoreLayersThanExistingChainLength() { assertThatThrownBy( () -> - trieLogHelper.prune( + nonValidatingTrieLogHelper.prune( dataStorageConfiguration, inMemoryWorldState, blockchain, Path.of(""))) .isInstanceOf(IllegalArgumentException.class) .hasMessage("Trying to retain more trie logs than chain length (5), skipping pruning"); @@ -227,13 +231,70 @@ public void cannotPruneIfUserRequiredFurtherThanFinalized(final @TempDir Path da assertThatThrownBy( () -> - trieLogHelper.prune( + nonValidatingTrieLogHelper.prune( dataStorageConfiguration, inMemoryWorldState, blockchain, dataDir)) .isInstanceOf(IllegalArgumentException.class) .hasMessage( "Trying to prune more layers than the finalized block height, skipping pruning"); } + @Test + public void skipPruningIfTrieLogCountIsLessThanMaxLayersToLoad() { + + DataStorageConfiguration dataStorageConfiguration = + ImmutableDataStorageConfiguration.builder() + .dataStorageFormat(BONSAI) + .bonsaiMaxLayersToLoad(6L) + .unstable( + ImmutableDataStorageConfiguration.Unstable.builder() + .bonsaiLimitTrieLogsEnabled(true) + .build()) + .build(); + + when(blockchain.getChainHeadBlockNumber()).thenReturn(5L); + + assertThatThrownBy( + () -> + nonValidatingTrieLogHelper.prune( + dataStorageConfiguration, inMemoryWorldState, blockchain, Path.of(""))) + .isInstanceOf(IllegalArgumentException.class) + .hasMessage("Trie log count (5) is less than retention limit (6), skipping pruning"); + } + + @Test + public void mismatchInPrunedTrieLogCountShouldNotDeleteFiles(final @TempDir Path dataDir) + throws IOException { + Files.createDirectories(dataDir.resolve("database")); + + DataStorageConfiguration dataStorageConfiguration = + ImmutableDataStorageConfiguration.builder() + .dataStorageFormat(BONSAI) + .bonsaiMaxLayersToLoad(3L) + .unstable( + ImmutableDataStorageConfiguration.Unstable.builder() + .bonsaiLimitTrieLogsEnabled(true) + .build()) + .build(); + + mockBlockchainBase(); + when(blockchain.getBlockHeader(5)).thenReturn(Optional.of(blockHeader5)); + when(blockchain.getBlockHeader(4)).thenReturn(Optional.of(blockHeader4)); + when(blockchain.getBlockHeader(3)).thenReturn(Optional.of(blockHeader3)); + + final BonsaiWorldStateKeyValueStorage inMemoryWorldStateSpy = spy(inMemoryWorldState); + // force a different value the second time the trie log count is called + when(inMemoryWorldStateSpy.streamTrieLogKeys(3L + DEFAULT_BONSAI_TRIE_LOG_PRUNING_WINDOW_SIZE)) + .thenCallRealMethod() + .thenReturn(Stream.empty()); + assertThatThrownBy( + () -> + nonValidatingTrieLogHelper.prune( + dataStorageConfiguration, inMemoryWorldStateSpy, blockchain, dataDir)) + .isInstanceOf(RuntimeException.class) + .hasMessage( + "Remaining trie logs (0) did not match --bonsai-historical-block-limit (3). Trie logs backup files have not been deleted, it is safe to rerun the subcommand."); + } + @Test public void trieLogRetentionLimitShouldBeAboveMinimum() { @@ -319,7 +380,7 @@ public void exceptionWhileSavingFileStopsPruneProcess(final @TempDir Path dataDi assertThatThrownBy( () -> - trieLogHelper.prune( + nonValidatingTrieLogHelper.prune( dataStorageConfiguration, inMemoryWorldState, blockchain, @@ -342,13 +403,13 @@ public void exceptionWhileSavingFileStopsPruneProcess(final @TempDir Path dataDi @Test public void exportedTrieMatchesDbTrieLog(final @TempDir Path dataDir) throws IOException { - trieLogHelper.exportTrieLog( + nonValidatingTrieLogHelper.exportTrieLog( inMemoryWorldState, singletonList(blockHeader1.getHash()), dataDir.resolve("trie-log-dump")); var trieLog = - trieLogHelper + nonValidatingTrieLogHelper .readTrieLogsAsRlpFromFile(dataDir.resolve("trie-log-dump").toString()) .entrySet() .stream() @@ -362,13 +423,13 @@ public void exportedTrieMatchesDbTrieLog(final @TempDir Path dataDir) throws IOE @Test public void exportedMultipleTriesMatchDbTrieLogs(final @TempDir Path dataDir) throws IOException { - trieLogHelper.exportTrieLog( + nonValidatingTrieLogHelper.exportTrieLog( inMemoryWorldState, List.of(blockHeader1.getHash(), blockHeader2.getHash(), blockHeader3.getHash()), dataDir.resolve("trie-log-dump")); var trieLogs = - trieLogHelper + nonValidatingTrieLogHelper .readTrieLogsAsRlpFromFile(dataDir.resolve("trie-log-dump").toString()) .entrySet() .stream() @@ -389,13 +450,14 @@ public void importedTrieLogMatchesDbTrieLog(final @TempDir Path dataDir) throws new BonsaiWorldStateKeyValueStorage( tempStorageProvider, new NoOpMetricsSystem(), DataStorageConfiguration.DEFAULT_CONFIG); - trieLogHelper.exportTrieLog( + nonValidatingTrieLogHelper.exportTrieLog( inMemoryWorldState, singletonList(blockHeader1.getHash()), dataDir.resolve("trie-log-dump")); var trieLog = - trieLogHelper.readTrieLogsAsRlpFromFile(dataDir.resolve("trie-log-dump").toString()); + nonValidatingTrieLogHelper.readTrieLogsAsRlpFromFile( + dataDir.resolve("trie-log-dump").toString()); var updater = inMemoryWorldState2.updater(); trieLog.forEach((k, v) -> updater.getTrieLogStorageTransaction().put(k, v)); @@ -413,13 +475,14 @@ public void importedMultipleTriesMatchDbTrieLogs(final @TempDir Path dataDir) th new BonsaiWorldStateKeyValueStorage( tempStorageProvider, new NoOpMetricsSystem(), DataStorageConfiguration.DEFAULT_CONFIG); - trieLogHelper.exportTrieLog( + nonValidatingTrieLogHelper.exportTrieLog( inMemoryWorldState, List.of(blockHeader1.getHash(), blockHeader2.getHash(), blockHeader3.getHash()), dataDir.resolve("trie-log-dump")); var trieLog = - trieLogHelper.readTrieLogsAsRlpFromFile(dataDir.resolve("trie-log-dump").toString()); + nonValidatingTrieLogHelper.readTrieLogsAsRlpFromFile( + dataDir.resolve("trie-log-dump").toString()); var updater = inMemoryWorldState2.updater(); trieLog.forEach((k, v) -> updater.getTrieLogStorageTransaction().put(k, v)); diff --git a/ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/engine/AbstractEngineNewPayload.java b/ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/engine/AbstractEngineNewPayload.java index a5b8acfc056..a9336497a1b 100644 --- a/ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/engine/AbstractEngineNewPayload.java +++ b/ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/engine/AbstractEngineNewPayload.java @@ -234,9 +234,12 @@ public JsonRpcResponse syncResponse(final JsonRpcRequestContext requestContext) return respondWithInvalid(reqId, blockParam, null, getInvalidBlockHashStatus(), errorMessage); } + final var blobTransactions = + transactions.stream().filter(transaction -> transaction.getType().supportsBlob()).toList(); + ValidationResult blobValidationResult = validateBlobs( - transactions, + blobTransactions, newBlockHeader, maybeParentHeader, maybeVersionedHashes, @@ -302,7 +305,8 @@ public JsonRpcResponse syncResponse(final JsonRpcRequestContext requestContext) final BlockProcessingResult executionResult = mergeCoordinator.rememberBlock(block); if (executionResult.isSuccessful()) { - logImportedBlockInfo(block, (System.currentTimeMillis() - startTimeMs) / 1000.0); + logImportedBlockInfo( + block, blobTransactions.size(), (System.currentTimeMillis() - startTimeMs) / 1000.0); return respondWith(reqId, blockParam, newBlockHeader.getHash(), VALID); } else { if (executionResult.causedBy().isPresent()) { @@ -380,10 +384,6 @@ JsonRpcResponse respondWithInvalid( invalidStatus, latestValidHash, Optional.of(validationError))); } - protected boolean requireTerminalPoWBlockValidation() { - return false; - } - protected EngineStatus getInvalidBlockHashStatus() { return INVALID; } @@ -396,15 +396,12 @@ protected ValidationResult validateParameters( } protected ValidationResult validateBlobs( - final List transactions, + final List blobTransactions, final BlockHeader header, final Optional maybeParentHeader, final Optional> maybeVersionedHashes, final ProtocolSpec protocolSpec) { - var blobTransactions = - transactions.stream().filter(transaction -> transaction.getType().supportsBlob()).toList(); - final List transactionVersionedHashes = new ArrayList<>(); for (Transaction transaction : blobTransactions) { var versionedHashes = transaction.getVersionedHashes(); @@ -489,7 +486,7 @@ private Optional> extractVersionedHashes( .collect(Collectors.toList())); } - private void logImportedBlockInfo(final Block block, final double timeInS) { + private void logImportedBlockInfo(final Block block, final int blobCount, final double timeInS) { final StringBuilder message = new StringBuilder(); message.append("Imported #%,d / %d tx"); final List messageArgs = @@ -503,9 +500,10 @@ private void logImportedBlockInfo(final Block block, final double timeInS) { message.append(" / %d ds"); messageArgs.add(block.getBody().getDeposits().get().size()); } - message.append(" / base fee %s / %,d (%01.1f%%) gas / (%s) in %01.3fs. Peers: %d"); + message.append(" / %d blobs / base fee %s / %,d (%01.1f%%) gas / (%s) in %01.3fs. Peers: %d"); messageArgs.addAll( List.of( + blobCount, block.getHeader().getBaseFee().map(Wei::toHumanReadableString).orElse("N/A"), block.getHeader().getGasUsed(), (block.getHeader().getGasUsed() * 100.0) / block.getHeader().getGasLimit(), diff --git a/ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/engine/EngineNewPayloadV1.java b/ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/engine/EngineNewPayloadV1.java index d7019543c02..9675fb01195 100644 --- a/ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/engine/EngineNewPayloadV1.java +++ b/ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/engine/EngineNewPayloadV1.java @@ -50,11 +50,6 @@ public String getName() { return RpcMethod.ENGINE_NEW_PAYLOAD_V1.getMethodName(); } - @Override - protected boolean requireTerminalPoWBlockValidation() { - return true; - } - @Override protected EngineStatus getInvalidBlockHashStatus() { return INVALID_BLOCK_HASH;