Create JSONL-based verbatim manifest format

[nadove-ucsc]

#2693 (comment)

As an intermediary step before implementing a full handover using the PFB format.

[hannes-ucsc]

For demo, generate a largish unfiltered JSONL manifest in anvilprod and—when the PR lands there— in prod. Show that every replica is listed exactly once. Show that the number of file and project/dataset replicas matches the count reported by the summary endpoint.

In anvilprod, generate a small JSONL manifest with filters matching files from both MA and non-public sources. Generate that manifest while providing the access token of a user that has access to all those sources. Generate that manifest again without providing an access token. Compare the manifests and show that the one generated anonymously does not contain entities from managed-access datasets.

[nadove-ucsc]

Generating an unfiltered manifest of the anvil5 catalog fails due to the lambda timing out after 30s while computing the content hash. There are 981,284 bundles in anvil5 but only 206,132 in dcp35.

[nadove-ucsc]

The largest manifest I was able to generate for anvil5 took about 5 minutes to generate, contained 1,202,647 replicas and was 507 MB. The lambda function's ephemeral storage is 512 MB, and larger manifests failed with the step-function error message: "OSError: [Errno 28] No space left on device".

[nadove-ucsc]

With the manifest content hash replaced with a hardcoded value and ephemeral storage increased to 1 GB, I ran into a new limit when trying to create an unfiltered manifest: the lambda timed out after 15 minutes. This is the maximum possible timeout.