Create file system based BlobSidecar data archive #8674
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
storage/src/test/java/tech/pegasys/teku/storage/archive/fsarchive/FileSystemArchiveTest.java
Fixed
Show fixed
Hide fixed
storage/src/main/java/tech/pegasys/teku/storage/archive/fsarchive/BlobSidecarJsonWriter.java
Outdated
Show resolved
Hide resolved
storage/src/main/java/tech/pegasys/teku/storage/server/kvstore/KvStoreDatabase.java
Outdated
Show resolved
Hide resolved
| @@ -170,6 +170,16 @@ public class BeaconNodeDataOptions extends ValidatorClientDataOptions { | |||
| arity = "0..1") | |||
| private int blobsPruningLimit = StorageConfiguration.DEFAULT_BLOBS_PRUNING_LIMIT; | |||
|
|
|||
| @CommandLine.Option( | |||
| names = {"--Xdata-storage-archive-path"}, | |||
There was a problem hiding this comment.
I'd add blob to the name of this flag to make it explicit that it is only a functionality that archive blobs rather than general (which gives the impression that it could store blocks as well as other things like state...
There was a problem hiding this comment.
Good call. I think you're looking at an older version. I did update this to "Xdata-storage-blobs-archive-path" to align with the other blobs options. :)
| final DataArchive dataArchive = | ||
| config | ||
| .getBlobsArchivePath() | ||
| .map(path -> (DataArchive) new FileSystemArchive(Path.of(path))) |
There was a problem hiding this comment.
you can avoid cast via
.<DataArchive>map(path -> new FileSystemArchive(Path.of(path)))
| import tech.pegasys.teku.spec.datastructures.blobs.versions.deneb.BlobSidecar; | ||
|
|
||
| /** | ||
| * Interface for a data archive which stores non-current BlobSidecars and could be extended later to |
There was a problem hiding this comment.
non-current, maybe something more specific like prunable or BlobSidecars outside the Data Availability window
|
|
||
| // Just warn if we failed to find all the BlobSidecars. | ||
| if (keys.size() != blobSidecars.size()) { | ||
| LOG.warn("Failed to retrieve BlobSidecars for keys:{}", keys); |
| if (!blobSidecarArchived) { | ||
| LOG.warn("Failed to archive BlobSidecar for slot:{}. Stopping pruning", key.getSlot()); | ||
| LOG.warn("Failed to archive and prune BlobSidecars. Stopping pruning"); |
There was a problem hiding this comment.
I wonder if this is an error. It requires an action from the user (like checking diskspace). We could also ask user to do something
| logPruningResult( | ||
| "Non canonical Blobs pruning finished in {} ms. Limit reached: {}", | ||
| nonCanonicalBlobsPruningStart, | ||
| nonCanonicalBlobsLimitReached); | ||
| } | ||
| } catch (ShuttingDownException | RejectedExecutionException ex) { | ||
| } catch (ShuttingDownException | RejectedExecutionException | IOException ex) { |
There was a problem hiding this comment.
IOException -> shutting down?
PR Description
Follows on from #8640 to allow pruned BlobSidecars to be written to the file system as an archive.
Initial implementation requires a base path where BlobSidecars are written to a directory structure as
basePath/ab/cd/abcd... where the hex string of the block root starts with abcd. The slot and block root
are also written to index.dat in the basePath directory.
Fixed Issue(s)
partially address #8513
Documentation
doc-change-requiredlabel to this PR if updates are required.Changelog