Add an option to avoid schedule receipt check immediately

[Chengxuan]

Currently, when a new transaction is registered for receipt, the confirmation manager schedules the check immediately. This caused two problems:

• If the transactions are new, the first receipt check is almost guaranteed to be wasted as the transaction wouldn't have been mined into a block yet.
• if the receipt was retrieve, duplicate receipt checks are possible due to a racing condition with the logic that schedules transaction receipt in processBlock. 8e15e84 demonstrates a naive fix from a different angle, but it won't work with forking situation.

The proposed PR delay the first fetch to when a new block is received or when the transaction has queued for longer than stale timeout. The behavior is put under a new configuration.

[peterbroadhurst]

@Chengxuan - question before approval here... what about the case where the block has already been mined by the time we add the transaction to the list.

This is concretely possible I think.

One example would be unlucky timing (particularly on a mine-on-demand chain).
Another would be if we've been down for a good period of time and are recovering transactions from the DB.

This I believe was the original reason for having the immediate check.

I don't disagree with removing it, but I wonder if the stale timeout is actually much too long to wait for a block to arrive.

[peterbroadhurst]

I think the default of false here would fine if it seems like the current behavior is undesirable in all use cases we're aware of.

We have some other big changes queued up for FFTM, so I feel like we should consider adding some notes to the next release. Maybe ensuring that there's a rollup of some of those notes to FireFly 1.3.1 when it's released.

[Chengxuan]

I agree, will make the change

[Chengxuan]

@peterbroadhurst in that case, the unscheduled transactions will be scheduled to get receipt whenever one of the following event happens:

1. the first time when any new block is detected (all unscheduled txs will get scheduled at this point to avoid missing historical txs that won't be mined in new blocks ever).
2. transaction has been in the queue for longer than staleReceiptTimeout
3. the block that contains the unscheduled transition is detected

In other words, for a subset of transactions (should be relatively small unless lots of transactions are not stuck in the tx pool on the node) "checking too early" situation. I don't think the "checking too early" situation can be avoided fully due to the nature of the problem.

[peterbroadhurst]

Thanks for the extra detail.

Give more to think about in terms of efficiency, but makes it clear this PR is great step forwards.