document nuances of replayDeletion: true in Usage API

[jbw976]

What's Missing?

The Usage API has been promoted to Beta and is on by default in v1.19.0. We had hoped to set replayDeletion: true as the default because of the performance boost during resource cleanup, but after discovering some potentially damaging and undesirable behavior, we decided to not make this the default. @turkenh captured this decision and behavior in:
crossplane/crossplane#6140 (comment)

We should capture this in the Usage docs page, so users can make an informed decision about when to use replayDeletion in their environment.
https://docs.crossplane.io/master/concepts/usages/#replay-blocked-deletion-attempt

[bobh66]

I wonder if replayDeletion only makes sense when there is a by resource specified? In that case there is generally a composite resource that is already marked for deletion and the GC is trying to delete it's children and is being blocked by the Usage webhook. replyDeletion is attempting to speed up this process by not waiting for the GC to retry deletion, since it may have backed off to the maximum delay (1000 seconds if I recall). If we didn't replay the deletion then GC would eventually do it anyway, which is completely out of anyone's control because the parent composite is marked deleted.

In the case where a Usage has no by attribute and is solely being used to prevent deletion of a resource, there is no parent composite that would be marked deleted and cause GC to eventually retry the deletion. So maybe it doesn't make sense to allow replayDeletion at all when there is no by specified? @turkenh ?

[bobh66]

This could even be taken one step further and we could check for an ownerReference on the Usage and follow it to see if the owner is marked deleted. If it is then replayDeletion makes sense, but if it isn't or there is no owner then it probably doesn't.

[bobh66]

I created crossplane/crossplane#6279 to consider restricting the application of replayDeletion