[Bug]: panic in soak test

[Sohill-Patel]

Problem encountered

https://jenkins.vega.rocks/job/common/job/system-tests-wrapper/59258/artifact/soak-test/err-node-7379.log

goroutine 232 [running]:
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).Run.func1.1()
	/jenkins/workspace/common/system-tests-wrapper/vega/cmd/vega/commands/node/node.go:166 +0x54
panic({0x5262880?, 0xc00110ac10?})
	/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/cometbft/cometbft/consensus.assertAppHashEqualsOneFromBlock({0xc002ba92c0, 0x20, 0x20}, 0xc0029c12c0)
	/jenkins/GOPATH/pkg/mod/github.com/cometbft/cometbft@v0.38.7/consensus/replay.go:546 +0x105
github.com/cometbft/cometbft/consensus.(*Handshaker).replayBlocks(_, {_, _}, {{{0xb, 0x1}, {0xc0063dd950, 0x6}}, {0xc0063dd960, 0xb}, 0x1, ...}, ...)
	/jenkins/GOPATH/pkg/mod/github.com/cometbft/cometbft@v0.38.7/consensus/replay.go:499 +0x23d
github.com/cometbft/cometbft/consensus.(*Handshaker).ReplayBlocksWithContext(_, {_, _}, {{{0xb, 0x1}, {0xc0063dd950, 0x6}}, {0xc0063dd960, 0xb}, 0x1, ...}, ...)
	/jenkins/GOPATH/pkg/mod/github.com/cometbft/cometbft@v0.38.7/consensus/replay.go:410 +0x44d
github.com/cometbft/cometbft/consensus.(*Handshaker).HandshakeWithContext(0xc0061dcc90, {0x6551568, 0x8d2a3c0}, {0x657e768?, 0xc00498c2a0?})
	/jenkins/GOPATH/pkg/mod/github.com/cometbft/cometbft@v0.38.7/consensus/replay.go:274 +0x478
github.com/cometbft/cometbft/node.doHandshake({_, _}, {_, _}, {{{0xb, 0x1}, {0xc0063dd950, 0x6}}, {0xc0063dd960, 0xb}, ...}, ...)
	/jenkins/GOPATH/pkg/mod/github.com/cometbft/cometbft@v0.38.7/node/setup.go:182 +0x1a9
github.com/cometbft/cometbft/node.NewNodeWithContext({0x6551568, 0x8d2a3c0}, 0xc001412280, {0x65428c0, 0xc0042410e0}, 0xc005424380, {0x651b800, 0xc006456a68}, 0xc005406901?, 0x6023690, ...)
	/jenkins/GOPATH/pkg/mod/github.com/cometbft/cometbft@v0.38.7/node/node.go:359 +0x5b7
github.com/cometbft/cometbft/node.NewNode(0x0?, {0x65428c0?, 0xc0042410e0?}, 0xa31a013802b13062?, {0x651b800?, 0xc006456a68?}, 0x6353797469646975?, 0x3538313a7365726f?, 0x3932323231316164?, {0x65516b8, ...}, ...)
	/jenkins/GOPATH/pkg/mod/github.com/cometbft/cometbft@v0.38.7/node/node.go:279 +0xa9
code.vegaprotocol.io/vega/core/blockchain/abci.NewTmNode({{0x0}, 0x1, 0x1, 0x0, 0x0, {0xc000d026b0, 0xa}, {{0x0}, {0xc000d0e138, 0x15}}, ...}, ...)
	/jenkins/workspace/common/system-tests-wrapper/vega/core/blockchain/abci/tm_node.go:100 +0x70a
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).startABCI(0xc000a54a00, 0x30?, {0x657e438, 0xc001408000}, {0x7ffcfbc75023, 0x53}, {0x0?, 0x28?}, {0x0, 0x0})
	/jenkins/workspace/common/system-tests-wrapper/vega/cmd/vega/commands/node/node.go:437 +0x18a
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).startBlockchain(0xc000a54a00, 0x6551488?, {0x7ffcfbc75023?, 0xc000d71f28?}, {0x0, 0x0}, {0x0, 0x0})
	/jenkins/workspace/common/system-tests-wrapper/vega/cmd/vega/commands/node/node.go:325 +0x39a
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).Run.func1()
	/jenkins/workspace/common/system-tests-wrapper/vega/cmd/vega/commands/node/node.go:169 +0x88
created by code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).Run in goroutine 1
	/jenkins/workspace/common/system-tests-wrapper/vega/cmd/vega/commands/node/node.go:160 +0x79d

Observed behaviour

soak-test failed on ST nightly with error
restore failed by falling out of consensus at a CHECKPOINT at height 7386, state: checkpoint-fail-7379-7386.json

Expected behaviour

soak-test to pass

Steps to reproduce

1. system-test nightly run of run for `tests/[a-dA-D]*`

Software version

dev branch

Failing test

No response

Jenkins run

https://jenkins.vega.rocks/job/common/job/system-tests-wrapper/59258/

Configuration used

No response

Relevant log output

No response

[wwestgarth]

This failed again on the next overnight this time due to an issue with the ELS snapshot and AMMs:

                {
                   "partyId": "6fe7ae732d36f40c1474c9de7c719b414ef8ae68ddf1e5323e5b7c5f8885450a",
-                  "share": "0.2511195665570905",
-                  "suppliedStake": "48811711099740000000000000",
-                  "virtualStake": "48811711099740000000000000",
-                  "avg": "128131352349547500000000000"
+                  "share": "0.3098585727534573",
+                  "suppliedStake": "67116468124500000000000000",
+                  "virtualStake": "67116468124500000000000000",
+                  "avg": "146436109374307500000000000"
                },