[Node Operator Issue] Granite Release bricks node, full re-sync required

[moorow]

Discussed in #564

^{Originally posted by moorow August 30, 2024}

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

Updating to the Granite release seems to break the state of a node.

Here are steps I followed and logs:

1. Graceful stop op-node

2. Graceful stop op-geth

3. Start op-geth (New version)

INFO [08-30|09:27:51.999] Post-Merge hard forks (timestamp based):
INFO [08-30|09:27:51.999]  - Shanghai:                    @1704992401 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [08-30|09:27:51.999]  - Cancun:                      @1710374401 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md)
INFO [08-30|09:27:51.999]  - Regolith:                    @0
INFO [08-30|09:27:51.999]  - Canyon:                      @1704992401
INFO [08-30|09:27:51.999]  - Ecotone:                     @1710374401
INFO [08-30|09:27:51.999]  - Fjord:                       @1720627201
INFO [08-30|09:27:51.999]  - Granite:                     @1726070401
INFO [08-30|09:27:51.999]
INFO [08-30|09:27:51.999] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [08-30|09:27:51.999]
INFO [08-30|09:27:52.000] Loaded most recent local block           number=124,705,593 hash=953dd8..ba9ede td=210,470,125 age=1m49s
INFO [08-30|09:27:52.000] Loaded most recent local finalized block number=124,704,852 hash=2c608b..db1b52 td=210,470,125 age=26m31s
INFO [08-30|09:27:52.000] Loaded last snap-sync pivot marker       number=123,961,326
WARN [08-30|09:27:52.000] Head state missing, repairing            number=124,705,593 hash=953dd8..ba9ede snaproot=2a2e50..f5c381
INFO [08-30|09:28:03.101] Block state missing, rewinding further   number=124,703,029 hash=baeb46..e4fa7a elapsed=11.100s
INFO [08-30|09:28:11.101] Block state missing, rewinding further   number=124,612,195 hash=441943..78c159 elapsed=19.100s
INFO [08-30|09:28:19.101] Block state missing, rewinding further   number=124,466,580 hash=de38e2..34caa6 elapsed=27.100s
INFO [08-30|09:28:27.101] Block state missing, rewinding further   number=124,251,092 hash=62041b..f1a58b elapsed=35.101s
INFO [08-30|09:28:35.101] Block state missing, rewinding further   number=124,005,258 hash=798f45..1b1643 elapsed=43.101s
INFO [08-30|09:28:36.505] Pivot block reached, resetting to genesis number=123,961,326 hash=5678e6..fd8b23
INFO [08-30|09:28:36.507] Chain is stateless, wait state sync      number=0           hash=7ca38a..3fa48b
INFO [08-30|09:28:36.508] Loaded most recent local header          number=124,705,593 hash=953dd8..ba9ede td=210,470,125 age=2m33s
INFO [08-30|09:28:36.508] Loaded most recent local block           number=0           hash=7ca38a..3fa48b td=1           age=55y5mo2w
INFO [08-30|09:28:36.508] Loaded most recent local snap block      number=124,705,593 hash=953dd8..ba9ede td=210,470,125 age=2m33s
INFO [08-30|09:28:36.508] Loaded most recent local finalized block number=124,704,852 hash=2c608b..db1b52 td=210,470,125 age=27m15s
INFO [08-30|09:28:36.508] Loaded last snap-sync pivot marker       number=123,961,326
WARN [08-30|09:28:36.509] Enabling snapshot recovery               chainhead=0 diskbase=124,705,308
WARN [08-30|09:28:36.526] Snapshot is not continuous with chain    snaproot=1a96d0..894acc chainroot=eddb4c..a568eb
INFO [08-30|09:28:36.526] Initialized transaction indexer          range="last 2350000 blocks"
INFO [08-30|09:28:36.526] Initialising Ethereum protocol           network=10         dbversion=8
INFO [08-30|09:28:36.526] Loaded local transaction journal         transactions=0 dropped=0
INFO [08-30|09:28:36.526] Enabled snap sync                        head=0 hash=7ca38a..3fa48b
INFO [08-30|09:28:36.526] Gasprice oracle is ignoring threshold set threshold=2
WARN [08-30|09:28:36.527] Engine API enabled                       protocol=eth

3. Start op-node (New version)

t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"

Protocol Description

network: op-mainnet
chainid: 10
L1 chain: mainnet

Old Versions

op-geth: v1.101315.2
op-node: v1.7.7

New Versions

op-geth: v1.101408.0
op-node: v1.9.1

Node Logs

No response

Additional Information

No response

Feedback

No response

[moorow]

Based on some discussion on Discord it looks like this may be related to Path Based (PBSS)?

Someone said that this only affects Full nodes, not Archive.

INFO [08-30|09:27:51.974] State schema set to default              scheme=path

This line was printed in the logs

[sbvegan]

Thanks for the flag, can you share your runtime configuration you're using for op-mode and op-geth?

[moorow]

Sure thing:

op-geth

      - --ws
      - --ws.port=8550
      - --ws.addr=0.0.0.0
      - --ws.origins="*"
      - --http
      - --http.port=8549
      - --http.addr=0.0.0.0
      - --http.vhosts="*"
      - --http.corsdomain="*"
      - --authrpc.addr=0.0.0.0
      - --authrpc.jwtsecret=/data/jwt.txt
      - --authrpc.port=9333
      - --authrpc.vhosts="*"
      - --datadir=/data/geth
      - --verbosity=3
      - --rollup.disabletxpoolgossip=true
      - --rollup.sequencerhttp=https://mainnet-sequencer.optimism.io
      - --op-network=op-mainnet
      - --syncmode=snap
      - --rpc.gascap=0
      - --rpc.evmtimeout=10s
      - --rpc.batch-response-max-size=50000000
      - --rpc.batch-request-limit=2000
      - --nat=extip:${HOST_IP_ADDRESS}

op-node

      - --l1=${L1_EXECUTION_RPC_URL}
      - --l1.rpckind=alchemy
      - --l1.trustrpc
      - --l1.beacon=${L1_BEACON_RPC_URL}
      - --l2=ws://op-geth:9333
      - --network=op-mainnet
      - --rpc.addr=127.0.0.1
      - --rpc.port=9545
      - --l2.jwt-secret=/data/jwt.txt
      - --syncmode=execution-layer
      - --p2p.advertise.ip=${HOST_IP_ADDRESS}
      - --p2p.listen.ip=0.0.0.0
      - --p2p.listen.tcp=9003
      - --p2p.listen.udp=9003
      - --p2p.advertise.tcp=9003
      - --p2p.advertise.udp=9003
      - --metrics.enabled
      - --metrics.addr=0.0.0.0
      - --metrics.port=7300

[sbvegan]

Thank you @moorow for surfacing this! We've added a State Schema Highlight in the op-geth release notes.

Per recently introduced changes from upstream geth (from the upstream Asteria release):

The default state trie representation is changed from hash mode to path mode (i.e. --state.scheme flipped from hash to path). This change does not affect op-geth instances with pre-existing databases, in the case of which op-geth continues to use whatever the existing database's format is. If no previous database exists however, for full nodes, op-geth will now default to pathdb. The main advantage is built-in, online historical state pruning; no more runaway state growth.

If you want to force the old behavior, you can run op-geth with --state.scheme=hash for now. That said, hash mode will most likely be dropped sooner rather than later, so we advise everyone running full nodes to gradually switch, if they haven't yet.
Archive mode support is not yet finalized for path mode, so archive nodes will still run in hash mode. Naturally, hash mode will not be dropped until a full path archive lands and people have enough time to switch to it.

[Eduard-Voiculescu]

@moorow Were you finally able to start a full node?

Did you do as @Staking7pc did and restart from a snapshot?

[Eduard-Voiculescu]

Ok I was able to finally run my node with the flag --state.scheme=hash. It rewinded much much further than the Granite Hard Fork that was what made me thing something was broken and not working as expected.

[moorow]

@Eduard-Voiculescu I ended up just doing a fresh re-sync. I think --state.scheme=hash is only a temporary solution as geth is moving away from hash towards path. Guessing hash may get fully deprecated at some stage.

[Eduard-Voiculescu]

Yep, I am doing the same. Afraid to see what will happen once they completely deprecate. I started post bedrock (from snapshot) which forces me to use hash state scheme...

[moorow]

Best thing to do is:

1. completely remove your data directory (no snapshot)
2. Make sure snapsync is enabled on op-geth and op-node (think it is by default now)
3. Set --state.scheme=path
4. ReSync from scratch, should take between 6-8 hours