Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dump p2p on files instead of hex logs #8177

Merged
merged 82 commits into from
Apr 17, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
4d3be14
Create P2PDumpManager with save invalid block functionality
courtneyeh Apr 7, 2024
13f1a8f
Create logic for other two file saves
courtneyeh Apr 8, 2024
8c3ef5d
Use unprefixed hex string in file name
courtneyeh Apr 8, 2024
164882e
Add testing
courtneyeh Apr 8, 2024
1fcfbd2
Add identifiers to file names
courtneyeh Apr 8, 2024
4184191
Rename variables
courtneyeh Apr 8, 2024
6659bad
Use arrival timestamp on message
courtneyeh Apr 8, 2024
e6e4b79
Use time provider for timestamp in test
courtneyeh Apr 8, 2024
087ecc9
Merge branch 'master' into p2p-dump
courtneyeh Apr 8, 2024
8b1bba7
Remove TODO
courtneyeh Apr 8, 2024
6521f6d
Merge branch 'master' into p2p-dump
courtneyeh Apr 9, 2024
3b6b567
Move P2PDumpManager
courtneyeh Apr 9, 2024
0abea63
Move P2PDumpManager to statetransition util
courtneyeh Apr 9, 2024
530c4bc
Initialise P2PDumpManager
courtneyeh Apr 9, 2024
3fbe240
Remove optional result
courtneyeh Apr 10, 2024
1f62c96
Merge branch 'master' into p2p-dump
courtneyeh Apr 10, 2024
68161a4
Spotless
courtneyeh Apr 10, 2024
7eb7cb5
Change return result
courtneyeh Apr 10, 2024
0ee2b96
Update P2PDumpManager
courtneyeh Apr 10, 2024
6099b66
Dump p2p on files instead of hex logs
courtneyeh Apr 10, 2024
a3eb42d
Fix tests and util
courtneyeh Apr 10, 2024
74669be
Merge branch 'master' into p2p-dump
courtneyeh Apr 10, 2024
91fccca
Highlighting issues
courtneyeh Apr 10, 2024
f6652af
Remove todos
courtneyeh Apr 10, 2024
882f3ae
Do not save to file if p2p warnings are disabled
courtneyeh Apr 10, 2024
c33baf2
Test not save to file when warnings disabled
courtneyeh Apr 10, 2024
f1ddf82
Update to new design
courtneyeh Apr 11, 2024
5abd7e0
Merge branch 'master' into p2p-dump
courtneyeh Apr 11, 2024
c6a9ea9
Update directory structure
courtneyeh Apr 11, 2024
d920c82
Fix referenceTestsPrep
courtneyeh Apr 11, 2024
239ee7e
Test new P2P Option
courtneyeh Apr 11, 2024
3079834
Add changelog entry
courtneyeh Apr 11, 2024
f6eafd9
Merge branch 'master' into p2p-dump
courtneyeh Apr 11, 2024
6062046
Update directory structure
courtneyeh Apr 11, 2024
c29ec81
Change manager name to DebugDataDumper
courtneyeh Apr 11, 2024
a82ea04
Few renames
courtneyeh Apr 11, 2024
08d78aa
Create NoOpDebugDataDumper
courtneyeh Apr 11, 2024
86b7e9e
Fix log inconsistency
courtneyeh Apr 11, 2024
4e7e077
Test not throw exception when unable to save to file
courtneyeh Apr 11, 2024
d871069
Use debug log
courtneyeh Apr 11, 2024
e214ad1
Remove identifiers
courtneyeh Apr 11, 2024
513c93e
Remove unused parameter
courtneyeh Apr 11, 2024
9ae5406
Fix description of p2p dump flag
courtneyeh Apr 11, 2024
0adaf95
Only create topic directories once
courtneyeh Apr 11, 2024
916d9ef
Add test for IO exception
courtneyeh Apr 11, 2024
0518042
Add disable breakers when failure and unable to return
courtneyeh Apr 11, 2024
e583fa0
Merge branch 'master' into p2p-dump
courtneyeh Apr 11, 2024
b732c5f
Test disabling service
courtneyeh Apr 11, 2024
26f295c
Test DebugDataDumper was used in ForkChoice
courtneyeh Apr 12, 2024
af98ec6
Test Eth2TopicHandler uses DebugDataDumper
courtneyeh Apr 12, 2024
d9b19f3
Update filenames
courtneyeh Apr 12, 2024
64770ff
Disable windows test
courtneyeh Apr 12, 2024
fa2bb78
Use mock debug data dumper in test
courtneyeh Apr 12, 2024
37cf086
Fix windows build
courtneyeh Apr 12, 2024
326f496
Format date in filename
courtneyeh Apr 12, 2024
1aff63f
Use timeprovider in test
courtneyeh Apr 12, 2024
70fb1f3
Small changes
courtneyeh Apr 12, 2024
25ee9d6
Rename for clarity
courtneyeh Apr 12, 2024
b2a3696
Remove static of timestamp formatter
courtneyeh Apr 12, 2024
c499511
Update simpler log info on file creation
courtneyeh Apr 12, 2024
7e2db7b
Handle unknown arrival time
courtneyeh Apr 12, 2024
33f63e2
Fix tests
courtneyeh Apr 12, 2024
9945286
Update test
courtneyeh Apr 12, 2024
69c7488
Fix windows build can't use : in filename
courtneyeh Apr 12, 2024
e82a751
Spotless
courtneyeh Apr 12, 2024
ae09a43
Fix unit test
courtneyeh Apr 12, 2024
689c9b7
Handle typical topic formatting
courtneyeh Apr 12, 2024
721e66d
Update success logging to be consistent with P2PLogger
courtneyeh Apr 13, 2024
fa338e6
Merge branch 'master' into p2p-dump
courtneyeh Apr 13, 2024
7c06149
Merge branch 'master' into p2p-dump
courtneyeh Apr 14, 2024
3a36a97
Merge branch 'master' into p2p-dump
courtneyeh Apr 15, 2024
3c17786
Move directory creation
courtneyeh Apr 15, 2024
6e402d6
Clean up timestamp file name
courtneyeh Apr 15, 2024
b4b7859
Rename tests
courtneyeh Apr 15, 2024
18a5fc6
Fix NoOpDebugDataDumper
courtneyeh Apr 15, 2024
92db86a
Fix test dirs
courtneyeh Apr 15, 2024
e7d6be9
Use NoOpDebugDataDumper when disabled in config
courtneyeh Apr 15, 2024
17e437e
Merge branch 'master' into p2p-dump
lucassaldanha Apr 16, 2024
1d3037d
Merge branch 'master' into p2p-dump
courtneyeh Apr 16, 2024
4247274
Pass block to data dumper
courtneyeh Apr 17, 2024
1f0bce0
Use supplier so message isn't calculated when disabled dumper
courtneyeh Apr 17, 2024
b5716d5
Merge branch 'master' into p2p-dump
lucassaldanha Apr 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions infrastructure/logging/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ dependencies {
implementation 'org.apache.logging.log4j:log4j-core'
compileOnly 'com.github.oshi:oshi-core'

testImplementation testFixtures(project(':ethereum:spec'))
courtneyeh marked this conversation as resolved.
Show resolved Hide resolved
testFixturesImplementation 'org.apache.logging.log4j:log4j-core'
testImplementation 'com.github.oshi:oshi-core'
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
/*
* Copyright Consensys Software Inc., 2024
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/

package tech.pegasys.teku.infrastructure.logging;

import java.io.IOException;
import java.nio.file.FileAlreadyExistsException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Optional;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.tuweni.bytes.Bytes;
import org.apache.tuweni.bytes.Bytes32;
import tech.pegasys.teku.infrastructure.unsigned.UInt64;

public class P2PDumpManager {
private static final Logger LOG = LogManager.getLogger();

private static final String INVALID_BLOCK_DIR = "invalid_blocks";

public static Optional<String> saveInvalidBlockToFile(
final Path directory, final UInt64 slot, final Bytes32 blockRoot, final Bytes blockSsz) {
// Check directory exists
final Path invalidBlocksDir = directory.resolve(INVALID_BLOCK_DIR);
if (invalidBlocksDir.toFile().mkdir()) {
LOG.info(INVALID_BLOCK_DIR + " directory has been created to save invalid blocks.");
}

final String fileName = String.format("slot%s_root%s.ssz", slot, blockRoot);
final Path path = invalidBlocksDir.resolve(fileName);

try {
// Create file and save ssz
if (!path.toFile().createNewFile()) {
final String errorMessage =
String.format(
"Unable to create new file to save invalid block. Slot: %s, Block Root: %s",
slot, blockRoot);
throw new FileAlreadyExistsException(errorMessage);
}
final Path writtenPath = Files.write(path, blockSsz.toArray());
return Optional.of(writtenPath.toString());

} catch (IOException e) {
final String errorMessage =
String.format(
"Failed to save invalid block bytes to file. Slot: %s, Block Root: %s",
slot, blockRoot);
LOG.error(errorMessage, e);
return Optional.empty(); // TODO What to do if error? Is returning empty enough?
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
/*
* Copyright Consensys Software Inc., 2024
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/

package tech.pegasys.teku.infrastructure.logging;

import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.fail;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Optional;
import org.apache.tuweni.bytes.Bytes;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import tech.pegasys.teku.spec.TestSpecFactory;
import tech.pegasys.teku.spec.datastructures.blocks.SignedBeaconBlock;
import tech.pegasys.teku.spec.util.DataStructureUtil;

class P2PDumpManagerTest {
Fixed Show fixed Hide fixed
final DataStructureUtil dataStructureUtil =
new DataStructureUtil(TestSpecFactory.createDefault());

@Test
void saveInvalidBlockToFile_shouldSaveToFile(@TempDir Path tempDir) {
final SignedBeaconBlock block = dataStructureUtil.randomSignedBeaconBlock();
final Optional<String> file =
P2PDumpManager.saveInvalidBlockToFile(
tempDir, block.getSlot(), block.getRoot(), block.sszSerialize());
assertThat(file).isPresent();

final Path expectedFile =
tempDir
.resolve("invalid_blocks")
.resolve(String.format("slot%s_root%s.ssz", block.getSlot(), block.getRoot()));
assertThat(file.get()).isEqualTo(expectedFile.toString());
checkBlockSavedToFile(expectedFile, block.sszSerialize());
}

@Test
void saveInvalidBlockToFile_failsToCreateFile(@TempDir Path tempDir) throws IOException {
final SignedBeaconBlock block = dataStructureUtil.randomSignedBeaconBlock();
final Path invalidBlocksDir = tempDir.resolve("invalid_blocks");

// Make invalid_blocks directory
assertThat(invalidBlocksDir.toFile().mkdir()).isTrue();

// Make file with expected name
final String fileName = String.format("slot%s_root%s.ssz", block.getSlot(), block.getRoot());
assertThat(invalidBlocksDir.resolve(fileName).toFile().createNewFile()).isTrue();

// Should not be able to create file when exists
final Optional<String> file =
P2PDumpManager.saveInvalidBlockToFile(
tempDir, block.getSlot(), block.getRoot(), block.sszSerialize());
assertThat(file).isEmpty();
checkBlockSavedToFile(invalidBlocksDir.resolve(fileName), Bytes.EMPTY);
}

private void checkBlockSavedToFile(final Path path, final Bytes expectedBytes) {
try {
final Bytes bytes = Bytes.wrap(Files.readAllBytes(path));
assertThat(bytes).isEqualTo(expectedBytes);
} catch (IOException e) {
fail();
}
}
}