p2p: failure to read complete message

[tony-iqlusion]

What went wrong?

There are ongoing reports of failure to read a complete message in the tendermint-p2p crate which seem to be manifesting as an incomplete Protobuf message in TMKMS:

iqlusioninc/tmkms#729

Steps to reproduce

Unfortunately this seems very chain-specific and I have not been given proper repro steps myself in the aforementioned issue.

Definition of "done"

It's unclear what actually needs to change here.

A somewhat related issue is the API that tendermint-p2p exposes: it's a message-oriented protocol, but the interface uses the std::io::{Read, Write} traits, which are stream-oriented. It's possible TMKMS is misusing the API, but it's the wrong abstraction to begin with.

[tony-iqlusion]

This might be related to #1356, although curiously people claim things work with Unix domain sockets and the errors only occur over TCP.

I did confirm read_exact is used: https://github.com/informalsystems/tendermint-rs/blob/64e9b33/p2p/src/secret_connection.rs#L622

[tony-iqlusion]

We're continuing to receive reports of this problem even with tendermint-p2p v0.34.1: iqlusioninc/tmkms#729 (comment)

[tony-iqlusion]

There's now a PR to tmkms to have it aggregate chunks of data itself and keep trying to decode them as a proto: iqlusioninc/tmkms#903

...and it really seems like tendermint-p2p should probably be doing that. Really I think the API should be refactored to be message-oriented, rather than stream-oriented.