DDP the codebase

[danieltudosiu]

Hey,

I am trying to use the codebase on a multi-gpu setup and I am running into the Data Parallelism limitations where the overhead on GPU 0 renders the parallelism's usefulness subpar at best.

Could you please enable DDP in the codebase? That could be achieved easily with Microsoft's Deepspeed library due to its automatic mpi discovery.

Cheer!

[lisun-ai]

Thanks for the suggestion. We will consider implementing the DDP, but it's not our priority right now. You are welcome to submit a pull request on this.