eth_getTransactionReceipt fails to find transaction

[nzenchik]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus fvm/fevm - Lotus FVM and FEVM interactions
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt/WinningPoSt)
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

Daemon:  1.25.2+mainnet+git.f51f83bfe+api1.5.0
Local: lotus version 1.25.2+mainnet+git.f51f83bfe

Repro Steps

1. Run 'curl -H "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params":["0xf9e71100c00934024a96b451b032c52e75ea46dfd3e50516eab6aba6afd569ad"],"id":1}''
2. See error '{"jsonrpc":"2.0","id":1,"error":{"code":1,"message":"failed to lookup Eth Txn 0xf9e71100c00934024a96b451b032c52e75ea46dfd3e50516eab6aba6afd569ad as bafy2bzaced46oeiayaetiasks22fdmbsyuxhl2sg37j6kbiw5k3kxjvp2vu22: failed to load message: ipld: could not find bafy2bzaced46oeiayaetiasks22fdmbsyuxhl2sg37j6kbiw5k3kxjvp2vu22"}}'

Describe the Bug

We are running archive node with EnableSplitstore = false, EnableEthRPC = true and EnableMsgIndex = true. For some transactions we get an error that they cannot be found. Here is an example
curl -H "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params":["0xf9e71100c00934024a96b451b032c52e75ea46dfd3e50516eab6aba6afd569ad"],"id":1}' {"jsonrpc":"2.0","id":1,"error":{"code":1,"message":"failed to lookup Eth Txn 0xf9e71100c00934024a96b451b032c52e75ea46dfd3e50516eab6aba6afd569ad as bafy2bzaced46oeiayaetiasks22fdmbsyuxhl2sg37j6kbiw5k3kxjvp2vu22: failed to load message: ipld: could not find bafy2bzaced46oeiayaetiasks22fdmbsyuxhl2sg37j6kbiw5k3kxjvp2vu22"}}
The same transaction on https://filfox.info/en/message/0xf9e71100c00934024a96b451b032c52e75ea46dfd3e50516eab6aba6afd569ad?t=1 shows that it has different ID (bafy2bzaceboq6gh6oztidkxg2aeac3tjsmpawqchdiji7ofpie57smofputrw) than returned by node

Logging Information

curl -H "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params":["0xf9e71100c00934024a96b451b032c52e75ea46dfd3e50516eab6aba6afd569ad"],"id":1}'
{"jsonrpc":"2.0","id":1,"error":{"code":1,"message":"failed to lookup Eth Txn 0xf9e71100c00934024a96b451b032c52e75ea46dfd3e50516eab6aba6afd569ad as bafy2bzaced46oeiayaetiasks22fdmbsyuxhl2sg37j6kbiw5k3kxjvp2vu22: failed to load message: ipld: could not find bafy2bzaced46oeiayaetiasks22fdmbsyuxhl2sg37j6kbiw5k3kxjvp2vu22"}}

[Stebalien]

So, this means that the translation between the message CID and the transaction hash isn't present in the index. I'm guessing that's happening because the node was restarted after it synced to a new head but before the indexer finished indexing.

Unfortunately, this code doesn't currently (although it really should...) record the last tipset indexed.

[Stebalien]

The fix is:

1. When we finish indexing the messages in a tipset, update some record of the last tipset indexed.
2. When we restart, look at the diff between the last tipset indexed and the first tipset reported by lotus, and index that diff.

[nzenchik]

Is there any way to fix this with current node state or the only way is to roll back head to block before this transaction and resync node from there?

[Stebalien]

Fixing this would require writing a bit of code to fill in the gaps.

[Stebalien]

Could you check if you have the message bafy2bzaceboq6gh6oztidkxg2aeac3tjsmpawqchdiji7ofpie57smofputrw on this node? Just to make sure?

[nzenchik]

Yes, node has this message

lotus chain getmessage bafy2bzaceboq6gh6oztidkxg2aeac3tjsmpawqchdiji7ofpie57smofputrw
{
  "Message": {
    "Version": 0,
    "To": "f410ftz6bnd4lzaquley4eeifleuxjbfqsefozmwd6nq",
    "From": "f410fhq3uzl2am6rmq6f6m5fxzdvhtisohkc7yowv27i",
    "Nonce": 4,
    "Value": "17000000000000000000",
    "GasLimit": 2278816,
    "GasFeeCap": "180234",
    "GasPremium": "110033",
    "Method": 3844450837,
    "Params": null,
    "CID": {
      "/": "bafy2bzacecvfyihe6iyinsciqfft3t4mchy73dwcooqmtw3maiopn6tcihxvm"
    }
  },
  "Signature": {
    "Type": 3,
    "Data": "NdvtU8Es+/amGUm0JkDQo5t/fc2srIQJqpIkt2lfWFYbaShWxQZCa1XCfkiS+frXxY82JMXo+IUHXCQ0UIOW4AA="
  },
  "CID": {
    "/": "bafy2bzaceboq6gh6oztidkxg2aeac3tjsmpawqchdiji7ofpie57smofputrw"
  }
}

[Stebalien]

Ok, then it's definitely an issue with the index.

[rjan90]

Hey! Just want to say that we are working on this issue as part of the larger meta issue: fixing high impact correctness and performance problems in ETH RPC API - please follow that issue ticket for further updates.

There is currently a WIP PR for the whole meta issue here: #12388