Valid blocks being marked as orphans and daemon getting out of sync

[JoaoFerreiraCLDF]

This happened already 3 times, in 3 different wallets from decredbrasil.com: Blocks start being marked as orphans and daemon gets out of sync.

Looking in the logs, it seems that one of the blocks is not received or processed and the following block, when received, is marked as orphan.

Here are some of the logs collected:

2020-06-24 17:51:11.461 [INF] BMGR: Processed 1 block in the last 46.37s (4 transactions, 0 tickets, 5 votes, 0 revocations, height 461198, 2020-06-24 17:51:08 +0000 UTC)
2020-06-24 17:54:48.087 [INF] CHAN: Adding orphan block 0000000000000000225e05e8f7d986ed7ca7f7ce687f1da53462638b88c33d36 with parent 00000000000000000ad1ab67778ebb90473e57f1e4151bbec657874bb03df7fc
2020-06-24 17:55:02.224 [INF] PEER: Peer 112.74.59.221:48984 (inbound) appears to be stalled or misbehaving, block timeout -- disconnecting
2020-06-24 17:55:02.225 [INF] BMGR: Lost peer 112.74.59.221:48984 (inbound)
2020-06-24 18:00:09.846 [INF] BMGR: New valid peer 45.63.16.87:46952 (inbound) (/decred-seeder:0.01/)
2020-06-24 18:00:11.013 [INF] BMGR: Lost peer 45.63.16.87:46952 (inbound)
2020-06-24 18:01:05.995 [INF] CHAN: Adding orphan block 00000000000000002405ac568cfb63dadee916126c406565a212a167559a77d7 with parent 0000000000000000225e05e8f7d986ed7ca7f7ce687f1da53462638b88c33d36
2020-06-24 18:03:37.654 [INF] BMGR: New valid peer 24.7.121.54:39300 (inbound) (/dcrwire:0.4.0/dcrd:1.5.1/)
2020-06-24 18:04:58.441 [INF] CHAN: Adding orphan block 0000000000000000103025b8681b90fba826350698b592b69540ab8010ead2b6 with parent 00000000000000002405ac568cfb63dadee916126c406565a212a167559a77d7

in that example, 00000000000000000ad1ab67778ebb90473e57f1e4151bbec657874bb03df7fc would have been block 461199, but it is not recognized as such by dcrd. Dcrd gets stuck in block 461198 until restarted.

2020-07-08 13:50:47.062 [INF] BMGR: Processed 1 block in the last 3m14.5s (4 transactions, 20 tickets, 5 votes, 1 revocation, height 465251, 2020-07-08 13:50:21 +0000 UTC)
2020-07-08 13:50:57.959 [INF] BMGR: New valid peer 101.86.123.241:35480 (inbound) (/dcrwire:0.3.0/dcrwallet:1.4.0-pre+dev/)
2020-07-08 13:52:46.251 [INF] BMGR: Processed 1 block in the last 1m59.18s (3 transactions, 20 tickets, 5 votes, 0 revocations, height 465252, 2020-07-08 13:52:07 +0000 UTC)
2020-07-08 13:54:22.373 [INF] BMGR: New valid peer 179.159.56.116:29338 (inbound) (/dcrwire:0.4.0/dcrd:1.5.1/)
2020-07-08 13:54:52.056 [INF] BMGR: Lost peer 101.86.123.241:35480 (inbound)
2020-07-08 13:56:00.040 [INF] BMGR: New valid peer 139.180.132.109:46168 (inbound) (/dcrwire:0.4.0/dcrd:1.5.0(rc2)/)
2020-07-08 13:56:04.929 [INF] BMGR: New valid peer 51.15.130.220:33682 (inbound) (/dcrwire:0.4.0/dcrd:1.5.1/)
2020-07-08 13:57:26.344 [INF] BMGR: New valid peer 54.175.255.216:41150 (inbound) (/dcrwire:0.4.0/dcrd:1.5.1/)
2020-07-08 13:57:48.927 [WRN] PEER: Peer 201.82.48.116:4202 (inbound) no answer for 5m0s -- disconnecting
2020-07-08 13:57:48.928 [INF] BMGR: Lost peer 201.82.48.116:4202 (inbound)
2020-07-08 13:58:18.446 [INF] BMGR: New valid peer 145.14.20.10:58868 (inbound) (/dcrwire:0.3.0/dcrwallet:1.4.0+release/)
2020-07-08 13:58:19.878 [INF] BMGR: Lost peer 145.14.20.10:58868 (inbound)
2020-07-08 14:00:09.481 [INF] BMGR: New valid peer 45.63.16.87:39407 (inbound) (/decred-seeder:0.01/)
2020-07-08 14:00:10.575 [INF] BMGR: Lost peer 45.63.16.87:39407 (inbound)
2020-07-08 14:00:37.085 [INF] BMGR: New valid peer 149.28.240.121:35996 (inbound) (/dcrwire:0.3.0/dcrpeersniffer:0.0.1/)
2020-07-08 14:00:37.319 [INF] BMGR: Lost peer 149.28.240.121:35996 (inbound)
2020-07-08 14:01:19.079 [INF] BMGR: New valid peer 45.63.43.126:48852 (inbound) (/dcrwire:0.4.0/dcrpeersniffer:0.0.1/)
2020-07-08 14:01:36.625 [INF] BMGR: Lost peer 45.63.43.126:48852 (inbound)
2020-07-08 14:02:21.672 [INF] BMGR: New valid peer 45.63.43.126:49294 (inbound) (/dcrwire:0.4.0/dcrpeersniffer:0.0.1/)
2020-07-08 14:02:40.107 [INF] BMGR: Lost peer 45.63.43.126:49294 (inbound)
2020-07-08 14:03:12.816 [INF] PEER: Peer 114.221.192.172:14809 (inbound) appears to be stalled or misbehaving, block timeout -- disconnecting
2020-07-08 14:03:12.816 [INF] BMGR: Lost peer 114.221.192.172:14809 (inbound)
2020-07-08 14:03:18.756 [INF] CHAN: Adding orphan block 00000000000000001e2cf5ec19fb9546ae480cef3a66bb980f1afc16d0265033 with parent 000000000000000005a33ae8154d9eeeeaa1af9d7aad3b6b9fce73d51e23a305

in this one, block 000000000000000005a33ae8154d9eeeeaa1af9d7aad3b6b9fce73d51e23a305 should have been 465253.

Restart solves the problem. Disconnecting the current "syncnode" with command dcrctl getpeerinfo | jq 'map(select(.syncnode==true))[] | .id' | dcrctl node disconnect - also works (as suggested by davecgh in slack).

[davecgh]

This is resolved by by #2555. Orphan blocks no longer exist as of that PR.