This PR shards the Dataloader across depth and data parallel ranks both #160

Sign in to view logs

Run time

Learn about OS pricing on GitHub Actions

Job	Run time
formatting / formatting	14s
nvidia-gpu / intra-layer	3m 8s
nvidia-gpu / inter-layer (1, 0)	3m 47s
nvidia-gpu / inter-layer (1, 1)	3m 20s
nvidia-gpu / inter-layer (2, 0)	5m 2s
nvidia-gpu / inter-layer (2, 1)	4m 51s
	20m 22s