Data Collection - takes too long

[cpranav93]

The current implementation of the data collection is taking a lot of runtime accounting for ~75% of the total runtime as per the profiler. This needs to be improved. Currently the following options are being considered:

• Zarr parallel storage
• Dask-zarr parallel storage
• Asyncio

From preliminary testing, zarr parallel storage is not showing any improvement but the use of dask/zarr significantly speeds up the model run (down from 180s to 50s).