Maintain per-project metadata JSONs

[hannes-ucsc]

┆Issue is synchronized with this Jira Story
┆Project Name: azul
┆Issue Number: AZUL-527

[diekhans]

Some empirical testing of Hannes' observation that updating the metadata aper-bundle event will be disastrously slow verifies this conclusion. While a simple increment metadata.json when adding a bundle can be linear, the kind of update that will happen when metadata is migrated is O(N^2) for building metadata.json. Empirically, O(N^2) is multiple days or even weeks with a single core.

Bundle events gives us 'map', but we don't have 'reduce'. Higher-level events are needed to do the 'reduce' phase.

Updating project-level data incrementally also creates a consistent-view problem of the project data. You may have a project that is partially metadata migrated that a user is seeing.

Until we have an architectural solution I would suggest we have the download client build metadata.json.

[theathorn]

Closing in favor of #2693.