[Node Operator Question] Granite Release bricks node, full re-sync required

[moorow]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

Updating to the Granite release seems to break the state of a node.

Here are steps I followed and logs:

1. Graceful stop op-node

2. Graceful stop op-geth

3. Start op-geth (New version)

INFO [08-30|09:27:51.999] Post-Merge hard forks (timestamp based):
INFO [08-30|09:27:51.999]  - Shanghai:                    @1704992401 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [08-30|09:27:51.999]  - Cancun:                      @1710374401 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md)
INFO [08-30|09:27:51.999]  - Regolith:                    @0
INFO [08-30|09:27:51.999]  - Canyon:                      @1704992401
INFO [08-30|09:27:51.999]  - Ecotone:                     @1710374401
INFO [08-30|09:27:51.999]  - Fjord:                       @1720627201
INFO [08-30|09:27:51.999]  - Granite:                     @1726070401
INFO [08-30|09:27:51.999]
INFO [08-30|09:27:51.999] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [08-30|09:27:51.999]
INFO [08-30|09:27:52.000] Loaded most recent local block           number=124,705,593 hash=953dd8..ba9ede td=210,470,125 age=1m49s
INFO [08-30|09:27:52.000] Loaded most recent local finalized block number=124,704,852 hash=2c608b..db1b52 td=210,470,125 age=26m31s
INFO [08-30|09:27:52.000] Loaded last snap-sync pivot marker       number=123,961,326
WARN [08-30|09:27:52.000] Head state missing, repairing            number=124,705,593 hash=953dd8..ba9ede snaproot=2a2e50..f5c381
INFO [08-30|09:28:03.101] Block state missing, rewinding further   number=124,703,029 hash=baeb46..e4fa7a elapsed=11.100s
INFO [08-30|09:28:11.101] Block state missing, rewinding further   number=124,612,195 hash=441943..78c159 elapsed=19.100s
INFO [08-30|09:28:19.101] Block state missing, rewinding further   number=124,466,580 hash=de38e2..34caa6 elapsed=27.100s
INFO [08-30|09:28:27.101] Block state missing, rewinding further   number=124,251,092 hash=62041b..f1a58b elapsed=35.101s
INFO [08-30|09:28:35.101] Block state missing, rewinding further   number=124,005,258 hash=798f45..1b1643 elapsed=43.101s
INFO [08-30|09:28:36.505] Pivot block reached, resetting to genesis number=123,961,326 hash=5678e6..fd8b23
INFO [08-30|09:28:36.507] Chain is stateless, wait state sync      number=0           hash=7ca38a..3fa48b
INFO [08-30|09:28:36.508] Loaded most recent local header          number=124,705,593 hash=953dd8..ba9ede td=210,470,125 age=2m33s
INFO [08-30|09:28:36.508] Loaded most recent local block           number=0           hash=7ca38a..3fa48b td=1           age=55y5mo2w
INFO [08-30|09:28:36.508] Loaded most recent local snap block      number=124,705,593 hash=953dd8..ba9ede td=210,470,125 age=2m33s
INFO [08-30|09:28:36.508] Loaded most recent local finalized block number=124,704,852 hash=2c608b..db1b52 td=210,470,125 age=27m15s
INFO [08-30|09:28:36.508] Loaded last snap-sync pivot marker       number=123,961,326
WARN [08-30|09:28:36.509] Enabling snapshot recovery               chainhead=0 diskbase=124,705,308
WARN [08-30|09:28:36.526] Snapshot is not continuous with chain    snaproot=1a96d0..894acc chainroot=eddb4c..a568eb
INFO [08-30|09:28:36.526] Initialized transaction indexer          range="last 2350000 blocks"
INFO [08-30|09:28:36.526] Initialising Ethereum protocol           network=10         dbversion=8
INFO [08-30|09:28:36.526] Loaded local transaction journal         transactions=0 dropped=0
INFO [08-30|09:28:36.526] Enabled snap sync                        head=0 hash=7ca38a..3fa48b
INFO [08-30|09:28:36.526] Gasprice oracle is ignoring threshold set threshold=2
WARN [08-30|09:28:36.527] Engine API enabled                       protocol=eth

3. Start op-node (New version)

t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"

Protocol Description

network: op-mainnet
chainid: 10
L1 chain: mainnet

Old Versions

op-geth: v1.101315.2
op-node: v1.7.7

New Versions

op-geth: v1.101408.0
op-node: v1.9.1

Node Logs

No response

Additional Information

No response

Feedback

No response

[hash1go]

Hello!
Since the continuation of this discussion (564) is taking place in that discussion (566), would it be alright to close this discussion (564)?