Reduce retention and early materialization of VirtualActionInput

[fmeum]

b82dcdc introduced a way for parameter files to be materialized lazily and in a streaming fashion. For this to be effective, the contents of VirtualActionInputs should not be retained and, ideally, never or only briefly materialized in full.

This is achieved by replacing the digest computations with a streaming approach and avoiding retention of VirtualActionInput in MerkleTrees, which may be cached. All cache protocols are streamlined to accept a Blob, which is a closeable supplier of InputStreams that is automatically closed after the upload has completed (which may require retries).

Along the way, file nodes in MerkleTrees are stored unwrapped to reduce memory usage.

[fmeum]

@justinhorvitz Context: I'm currently investigating higher memory usage with path mapping enabled and am thus planning to make it compatible with your command line preprocessing logic. While working on that, I noticed that Bazel may not make good use of this optimization as it's RE implementation may retain full param file contents and some other places would at least temporarily materialize the full file just to calculate a digest. This PR is meant to address that. Ignoring the specifics of the integration with Bazel's RE backend, which other folks can review, do you consider it reasonable?

[justinhorvitz]

Might virtualActionInput.getBytes() be called multiple times by Chunker? The approach we use for the Google-internal RE is:

• Materialize the ByteString as late as possible, using double-checked locking, storing it as a field in the "supplier"
• The "supplier" has a close() method that is called when all chunks have been sent to the CAS. Calling this nulls the ByteString field.

[justinhorvitz]

temporarily materialize the full file just to calculate a digest

This is a good catch. The google-internal RE code does not do this. Instead, it makes use of VirtualActionInput#writeTo, passing an OutputStream that computes the digest on the fly. It looks like that's the same approach you've gone with in this PR.

I left a comment about the way you're passing a supplier into Chunker. I think you have the right approach, just need to make sure that VirtualActionInput#getBytes is only called once.

[fmeum]

@bazel-io fork 8.1.0

[iancha1992]

@fmeum Missing a Javadoc comment in line 62 and 64 from src/main/java/com/google/devtools/build/lib/remote/merkletree/MerkleTree.java

[fmeum]

@fmeum Missing a Javadoc comment in line 62 and 64 from src/main/java/com/google/devtools/build/lib/remote/merkletree/MerkleTree.java

@iancha1992 Done!

[tjgq]

Note: I had to amend the VirtualActioninput#getBytes signature to throws IOException as we have Google-internal overrides with a throwing signature. (I suspect they all boil down to an in-memory operation that cannot throw, but there's too much indirection for me to easily fix it. Sorry.)