Concurrency Issues Diagnosis

[derrandz]

Description:

Blockchain reactor v1 gets stuck after node peering

Theory:

Messages and Errors channels on the reactors are becoming full and thus blocking send routines, and they keep blocking for as long as they are not attended to due to their priority.

Actuality:

This specific situation appeared on Max Priority / Full Channel instance

[derrandz]

Avenues to try:

1. Write integration tests using the node package node.go and find ways to reproduce by setting channel capacity and reactor capacity to reachable numbers for a test environment.

2. Run a development stack (3 or 5 nodes) with hot reload on tendermint code change, and simulate manually.

Favoring 2 over 1 until it is clear what we are dealing with

[derrandz]

Efforts relating to avenue (2) are available in the following PR