error when use single node and multi gpu for training

[lrj16]

I can train successfully by 1 node and 1 gpu, but when i use 1 node and 4 gpu, it trigger error——ValueError: you need to add an explicit nodesplitter to your input pipeline for multi-node training. Does any one know how to fix it?

[xvjiarui]

Hi @lrj16 , what's webdataset version? Could you please post the error trace?

[lrj16]

Hi @lrj16 , what's webdataset version? Could you please post the error trace?

@xvjiarui Thanks for replying. My webdataset version is 0.2.33. I have solved this problem. But this is my first time use webdataset, and I have a new problem. I need a more complex dataloader, which can return more information beside <image,text> pair. But map_dict(key=funct()) just can return the corresponding value of the key. I tried map(add_key()) to create new key but result in error as follow, do you know why? Thanks very much.

[lrj16]

@xvjiarui I have solved this problem. Thanks!

[lianxxx]

I can train successfully by 1 node and 1 gpu, but when i use 1 node and 4 gpu, it trigger error——ValueError: you need to add an explicit nodesplitter to your input pipeline for multi-node training. Does any one know how to fix it?

I had the same problem. How did you solve it? Thanks!!!

[lianxxx]

Hi @lrj16 , what's webdataset version? Could you please post the error trace?

Thanks Author, problem solved. Changed webdataset version to 0.1.103.