Different results for blocks between masternode and block service

[Endogen]

For querying block data (which is the most important one since it's the raw dataset) there are cases in which the block service delivers different data than the masternode.

Case 1)

For example for block 700923

Masternode - https://masternode-01.lamden.io/blocks?num=700923

{"hash": "31a9f4856cbdf1f5d6f6ae58195894f85680f597ac5cd1003f5f95ea97fb970f", "number": 700923, "previous": "005499cf99e16bf0af3f01ffd4ffe7e266285a56d5443be2d0bc4a56a8ff81a3", "subblocks": [{"input_hash": "0e2a8d4eba960f9fdff961563112cd3a1a584b10069c13e7d0e01fd4322825bc", "merkle_leaves": ["1f71a455ea3a4e2858b6bfd0c810e8cbca106baceaa4ae16b37650d35f9d84d1"], "signatures": [{"signature": "d0ae334b2cab7c81a85d61afba8b41d56ff381fefee821b0cc58247ddcc8c8aad10843644dcd71ce267e0948053be0611c1c242fc8927fdb5db8b71fbe150502", "signer": "2bdb7a98d65a469dacd93873ce3f8b6bb5284414348cab17ade738f913d35e32"}, {"signature": "778e911d1384310e3dcaeca38378cca5b13264947653070868ffa8297e6540efd50f94702be3db858fd7624a616459a2ca86f76514d781f8636a1f275c6d7903", "signer": "555100f02ef3bdc469e866140c35955e137624f4a1284bbcb999cd7fb576c869"}], "subblock": 0, "transactions": [{"hash": "f6c2fd1ac847ce057e2aa8b24356ab23f9cadfe3e0a321b69e497f2aec1aca77", "result": "None", "stamps_used": 103, "state": [{"key": "con_uw_master__s__1.balances:268f358cc188b7ad9f3307f61518e26b9d7ffeb455a9795dd3c63611f8ecabb2:a3510236c324b622eb2aee2b980fe35d39d6bc881cd77af19b8094d68c04f4eb:con_uw_evolve_t", "value": true}, {"key": "currency.balances:268f358cc188b7ad9f3307f61518e26b9d7ffeb455a9795dd3c63611f8ecabb2", "value": {"__fixed__": "92.004179016724608449432644600115"}}], "status": 0, "transaction": {"metadata": {"signature": "f22400854e37f64b5b51e38e7617c8859aebefa9cf95e0657be58ea9e8fb1133a944eec1e2aef3b2c1c9f650e4bfcf7698e966fc80fddaf8335117b9c2247b03", "timestamp": 1644436526}, "payload": {"contract": "con_uw_master__s__1", "function": "approve", "kwargs": {"contract": "con_uw_nft_crystal", "to": "con_uw_evolve_t", "uid": "a3510236c324b622eb2aee2b980fe35d39d6bc881cd77af19b8094d68c04f4eb"}, "nonce": 1252, "processor": "5b09493df6c18d17cc883ebce54fcb1f5afbd507533417fe32c006009a9c3c4a", "sender": "268f358cc188b7ad9f3307f61518e26b9d7ffeb455a9795dd3c63611f8ecabb2", "stamps_supplied": 200}}}]}]}

Block Service - http://119.29.130.37:3535/blocks/700923

{"error":"block number 700923 does not exist."}

Case 2)

It seems that some block services are not always returning an error if a block doesn't exist - which makes it hard to automatically detect that a block doesn't exist

For example for block 844646

Masternode - https://masternode-01.lamden.io/blocks?num=844646

{"error":"Block not found."}

Block Service for Nebula Finance:

{"hash":"block-does-not-exist","number":844646,"subblocks":[]}

In the case of this specific block service, there are cases where it delivers an error (for example block 651711) and cases where it delivers a "block-does-not-exist". It seems that the cases where it delivers an error are not errors at all (at least the masternode can deliver the block) and in the case of "block-does-not-exist" it looks like the block really is non-existant.

The expected behavior would be that block service and masternode agree on the state of all blocks and hold the same data and if a block doesn't exist, then there should be an error and not a hash with value block-does-not-exist because now we need to actually check the value of hash to know if a block exists or not.

In any case, the conclusion is that if you use the block service then it is not guaranteed that you know the correct global state which is disastrous.

[Dapiguabc]

@Endogen http://119.29.130.37:3535 is a demo of block service. Blocks data come from https://testnet-master-1.lamden.io. So... it's not a problem.

[Endogen]

@Endogen http://119.29.130.37:3535 is a demo of block service. Blocks data come from https://testnet-master-1.lamden.io. So... it's not a problem.

Just replace that block service with any other and you get the same result.

[Dapiguabc]

@Endogen http://119.29.130.37:3535 is a demo of block service. Blocks data come from https://testnet-master-1.lamden.io. So... it's not a problem.

Just replace that block service with any other and you get the same result.

take a look.
http://165.22.47.195:3535/blocks/700923 # mainnet
https://masternode-01.lamden.io/blocks?num=700923

[Endogen]

If different block services deliver different things then we still have an issue:
https://blockservice.nebulamden.finance/blocks/700923

And case 2) is also still an issue

[Endogen]

It's just inconsistent. The block service here gives me data for this block:
http://165.22.47.195:3535/blocks/761414

However another block service i can't expose (from the Rubix project) shows for the same block:

{"error":"block number 761414 does not exist."}

[Endogen]

So it seems the situation is even worse. We can't rely on block services having the same data. That's an issue since projects need to run their own block service and can't use a centralized one. If the block service has functionality included to correct missing blocks then it doesn't work. If it doesn't have that functionality then it needs to get it.

[Dapiguabc]

If different block services deliver different things then we still have an issue: https://blockservice.nebulamden.finance/blocks/700923

And case 2) is also still an issue

1. Syncing block data need take some time.
2. Maybe some blocks are missing and not be repaired. (It seems that the newest code of block service will check for missing block every 5 min)

These two reasons may lead to this issue.

I am not sure which version of blockservice the server https://blockservice.nebulamden.finance/blocks/700923 is using.

[Endogen]

Not having the newest version could definitely be the issue here, yes. How can i check which version i have?

[JeffWScott]

It's just inconsistent. The block service here gives me data for this block: http://165.22.47.195:3535/blocks/761414

However another block service i can't expose (from the Rubix project) shows for the same block:

{"error":"block number 761414 does not exist."}

Blockservices are not synced to each other. It's completely possible for one to not have digested a block another happened to get. They are seperate services run by the person or app that needs a block service.

In your example one block service didn't get a block, for some reason. That reason could be as simple as a network hic-up on the blockservice's end or the masternode went down and there was some network error. Who knows.

The short term fix for the blockservice owner is to run a repair. It's your blockservice and you need to maintain it, especially if you want to best uptime for your app.

There needs to be a more holistic fix for missing blocks. Where the blockservice can figure that out and proactively repair. That's something we should look into.

[Endogen]

What's the thing about getting blocks like this:

{"hash":"block-does-not-exist","number":507274,"subblocks":[]}

Is it possible to get rid of that? The masternode doesn't deliver that, yet there are blocks in the block service with that content.

[JeffWScott]

If different block services deliver different things then we still have an issue: https://blockservice.nebulamden.finance/blocks/700923
And case 2) is also still an issue

1. Syncing block data need take some time.
2. Maybe some blocks are missing and not be repaired. (It seems that the newest code of block service will check for missing block every 5 min)

These two reasons may lead to this issue.

I am not sure which version of blockservice the server https://blockservice.nebulamden.finance/blocks/700923 is using.

Yes, the blockservice is supposed to run some repair every once and a while. It doesn't seem to be working all that great as even the Wallet blockservice (which is the newest code) seems to also have missing blocks. I will proactively go run a repair on the Wallet blockservice manually from time to time.

@Dapiguabc can you look into the best was to approach this? If you want to Jam on it on Monday we can.

[JeffWScott]

https://masternode-01.lamden.io/blocks?num=507274

That is a block that doesn't infact exist on the blockchain. I think the data is appropriate.

[Endogen]

These are all blocks that don't exist - everything else exists:

70704
501400
501401
507274
507275
507992
626402
844631
844646

[Endogen]

https://masternode-01.lamden.io/blocks?num=507274

That is a block that doesn't infact exist on the blockchain. I think the data is appropriate.

But then deliver this data:

{"error":"Block not found."}

and not - http://165.22.47.195:3535/blocks/70704

{"hash":"block-does-not-exist","number":70704,"subblocks":[]}

[JeffWScott]

These are all blocks that don't exist - everything else exists:

70704
501400
501401
507274
507275
507992
626402
844631
844646

You know that. But the blockservice doesn't necessarily know that. If it queries a block and gets a network error or some other response due to a masternode issue then it needs to store something. In this case it stores that the block doest not exist and then will follow up later.

[Endogen]

These are all blocks that don't exist - everything else exists:

70704
501400
501401
507274
507275
507992
626402
844631
844646

You know that. But the blockservice doesn't necessarily know that. If it queries a block and gets a network error or some other response due to a masternode issue then it needs to store something. In this case it stores that the block doest not exist and then will follow up later.

Ahh ok. So that is what the block service delivers you if it couldn't get that block for some reason. OK got it.

[JeffWScott]

These are all blocks that don't exist - everything else exists:

70704
501400
501401
507274
507275
507992
626402
844631
844646

You know that. But the blockservice doesn't necessarily know that. If it queries a block and gets a network error or some other response due to a masternode issue then it needs to store something. In this case it stores that the block doest not exist and then will follow up later.

Ahh ok. So that is what the block service delivers you if it couldn't get that block for some reason. OK got it.

Yeah, repairing that info is really what is needed. Benji mentioned some logic to be months ago that I think might work. I'll follow up with him.