tools: add t8n.sh to the t8n-dump-dir for evmone, besu & ethereum-specs-evm (and various t8n tool clean-up)

docs/getting_started/debugging_t8n_tools.md

@@ -0,0 +1,101 @@
+# Debugging Transition Tools
+
+The `--t8n-dump-dir` flag can be used to dump the inputs and outputs of every call made to the `t8n` command to help debugging or simply understand how a test is interacting with the EVM.
+
+In particular, a script `t8n.sh` is generated for each call to the `t8n` command which can be used to reproduce the call to trigger errors or attach a debugger without the need to execute Python.
+
+For example, running:
+
+```console
+fill tests/berlin/eip2930_access_list/ --fork Berlin \
+    --t8n-dump-dir=/tmp/evm-t8n-dump
+```
+
+will produce the directory structure:
+
+```console
+/tmp/evm-t8n-dump/
+└── test_access_list_fork_Berlin
+    ├── 0
+    │   ├── args.py
+    │   ├── input
+    │   │   ├── alloc.json
+    │   │   ├── env.json
+    │   │   └── txs.json
+    │   ├── output
+    │   │   ├── alloc.json
+    │   │   ├── result.json
+    │   │   └── txs.rlp
+    │   ├── returncode.txt
+    │   ├── stderr.txt
+    │   ├── stdin.txt
+    │   ├── stdout.txt
+    │   └── t8n.sh
+    └── 1
+        ├── args.py
+        ├── input
+        │   ├── alloc.json
+        │   ├── env.json
+        │   └── txs.json
+        ├── output
+        │   ├── alloc.json
+        │   ├── result.json
+        │   └── txs.rlp
+        ├── returncode.txt
+        ├── stderr.txt
+        ├── stdin.txt
+        ├── stdout.txt
+        └── t8n.sh
+```
+
+where the directories `0` and `1` correspond to the different calls made to the `t8n` tool executed during the test, in this case (pre-Shanghai state):
+
+- `0` corresponds to the call used to calculate the state root of the test's initial alloc (which is why it has an empty transaction list).
+- `1` corresponds to the call used to execute the first transaction from the test.
+
+For post-Shanghai forks:
+
+- `0` corresponds to the call used to calculate the `stateRoot` for the test's initial alloc.
+- `1` (optional) corresponds to the call used to calculate the initial `withdrawalsRoot` for the test's initial alloc if required.
+- `2` corresponds to the call used to execute the first transaction from the test.
+
+Note, there may be more directories present `3`, `4`,... if the test executes more transactions.
+
+Each directory contains files containing information corresponding to the call, for example, the `args.py` file contains the arguments passed to the `t8n` command and the `output/alloc.json` file contains the output of the `t8n` command's `--output-alloc` flag.
+
+## The `t8n.sh` Script
+
+The `t8n.sh` script written to the debug directory can be used to reproduce any call made to the `t8n` command, for example, if a Besu `t8n-server` has been started on port `3001`, the request made by the test for first transaction can be reproduced as:
+
+```console
+/tmp/besu/test_access_list_fork_Berlin/1/t8n.sh 3021
+```
+
+which writes the response the from the `t8n-server` to the console output:
+
+```json
+{
+  "alloc" : {
+    "0x000000000000000000000000000000000000aaaa" : {
+      "code" : "0x5854505854",
+      "balance" : "0x4",
+      "nonce" : "0x1"
+    },
+    "0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba" : {
+      "balance" : "0x1bc16d674ecb26ce"
+    },
+    "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b" : {
+      "balance" : "0x2cd931",
+      "nonce" : "0x1"
+    }
+  },
+  "body" : "0xf8a0b89e01f89b0180078304ef0094000000000000000000000000000000000000aaaa0180f838f7940000000000000000000000000000000000000000e1a0000000000000000000000000000000000000000000000000000000000000000001a02e16eb72206c93c471b5894800495ee9c64ae2d9823bcc4d6adeb5d9d9af0dd4a03be6691e933a0816c59d059a556c27c6753e6ce76d1e357b9201865c80b28df3",
+  "result" : {
+    "stateRoot" : "0x51799508f764047aee6606bc6a00863856f83ee5b91555f00c8a3cbdfbec5acb",
+    ...
+    ...
+  }
+}
+```
+
+The `t8n.sh` is written to the debug directory for all [supported t8n tools](../index.md#transition-tool-support).

docs/getting_started/executing_tests_command_line.md

@@ -90,47 +90,7 @@ fill tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py::test_warm_coinb
 
 ## Debugging the `t8n` Command
 
-The `--t8n-dump-dir` flag can be used to dump the inputs and outputs of every call made to the `t8n` command for debugging purposes:
-
-```console
-fill --t8n-dump-dir=/tmp/evm-t8n-dump
-```
-
-For example, running:
-
-```console
-fill tests/berlin/eip2930_access_list/ --fork Berlin \
-    --t8n-dump-dir=/tmp/evm-t8n-dump
-```
-
-will produce the directory structure:
-
-```console
-/tmp/evm-t8n-dump/
-└── test_access_list_fork_Berlin
-    ├── 0
-    │   ├── alloc
-    │   ├── args
-    │   ├── env
-    │   ├── output_alloc
-    │   ├── output_result
-    │   ├── returncode
-    │   ├── stderr
-    │   ├── stdout
-    │   └── txs
-    └── 1
-        ├── alloc
-        ├── args
-        ├── env
-        ├── output_alloc
-        ├── output_result
-        ├── returncode
-        ├── stderr
-        ├── stdout
-        └── txs
-```
-
-where the directories `0` and `1` correspond to the different calls made to the `t8n` tool executed during the test. Each directory then contain files containing information corresponding to the call, for example, the `args` file contains the arguments passed to the `t8n` command and the `output_alloc` file contains the output of the `t8n` command's `--output-alloc` flag. Note, the first call is used to calculate the state root of the starting alloc and therefore has an empty transaction list.
+The `--t8n-dump-dir` flag can be used to dump the inputs and outputs of every call made to the `t8n` command for debugging purposes, see [Debugging Transition Tools](./debugging_t8n_tools.md).
 
 ## Other Useful Pytest Command-Line Options
 

docs/navigation.md

@@ -7,6 +7,7 @@
     * [Executing Tests at a Prompt](getting_started/executing_tests_command_line.md)
     * [Executing Tests in VS Code](getting_started/executing_tests_vs_code.md)
     * [Executing Tests for Features Under Development](getting_started/executing_tests_dev_fork.md)
+    * [Debugging Transition Tools](getting_started/debugging_t8n_tools.md)
   * [Writing Tests](writing_tests/index.md)
     * [Code Standards](writing_tests/code_standards.md)
     * [Types of Test](writing_tests/types_of_tests.md)
@@ -20,8 +21,9 @@
   * [Developer Doc](dev/index.md)
     * [Documentation](dev/docs.md)
     * [Coding Style](dev/coding_style.md)
+    * [Enabling Precommit Checks](dev/precommit.md)
   * [Library Reference](library/index.md)
     * [EVM Transition Tool Package](library/evm_transition_tool.md)
     * [Ethereum Test Tools Package](library/ethereum_test_tools.md)
     * [Ethereum Test Forks Package](library/ethereum_test_forks.md)
-    * [Pytest Plugins](library/pytest_plugins/)
+    * [Pytest Plugins](library/pytest_plugins/index.md)

src/evm_transition_tool/besu.py

@@ -2,8 +2,10 @@
 Hyperledger Besu Transition tool frontend.
 """
 
+import json
 import re
 import subprocess
+import textwrap
 from pathlib import Path
 from re import compile
 from typing import Any, Dict, List, Optional, Tuple
@@ -109,32 +111,67 @@ def evaluate(
             "chainid": chain_id,
             "reward": reward,
         }
+
+        post_data = {"state": state_json, "input": input_json}
+
         if debug_output_path:
+            post_data_string = json.dumps(post_data, indent=4)
+            additional_indent = " " * 16  # for pretty indentation in t8n.sh
+            indented_post_data_string = "{\n" + "\n".join(
+                additional_indent + line for line in post_data_string[1:].splitlines()
+            )
+            t8n_script = textwrap.dedent(
+                f"""\
+                #!/bin/bash
+                # Use $1 as t8n-server port if provided, else default to 3000
+                PORT=${{1:-3000}}
+                curl http://localhost:${{PORT}}/ -X POST -H "Content-Type: application/json" \\
+                --data '{indented_post_data_string}'
+                """
+            )
             dump_files_to_directory(
                 debug_output_path,
-                input_json
-                | {
-                    "state": state_json,
+                {
+                    "state.json": state_json,
+                    "input/alloc.json": input_json["alloc"],
+                    "input/env.json": input_json["env"],
+                    "input/txs.json": input_json["txs"],
+                    "t8n.sh+x": t8n_script,
                 },
             )
 
-        response = requests.post(
-            self.server_url,
-            json={
-                "state": state_json,
-                "input": input_json,
-            },
-            timeout=5,
-        )
+        response = requests.post(self.server_url, json=post_data, timeout=5)
         response.raise_for_status()  # exception visible in pytest failure output
         output = response.json()
 
         if debug_output_path:
             dump_files_to_directory(
                 debug_output_path,
                 {
-                    "output_alloc": output["alloc"],
-                    "output_result": output["result"],
+                    "response.txt": response.text,
+                    "status_code.txt": response.status_code,
+                    "time_elapsed_seconds.txt": response.elapsed.total_seconds(),
+                },
+            )
+
+        if response.status_code != 200:
+            raise Exception(
+                f"t8n-server returned status code {response.status_code}, "
+                f"response: {response.text}"
+            )
+        if not all([x in output for x in ["alloc", "result", "body"]]):
+            raise Exception(
+                "Malformed t8n output: missing 'alloc', 'result' or 'body', server response: "
+                f"{response.text}"
+            )
+
+        if debug_output_path:
+            dump_files_to_directory(
+                debug_output_path,
+                {
+                    "output/alloc.json": output["alloc"],
+                    "output/result.json": output["result"],
+                    "output/txs.rlp": output["body"],
                 },
             )