OOM error

[Davido111200]

Hi,

Thanks for sharing this awesome repo. I'm having memory problem with running run.sh with 1 H100 gpu. I tried reducing batch sizes to 1, but still I can not fine-tune a 7B LLM. Any idea how to work this around?

[Hprairie]

Hey did you ever get around this? How much memory was required? I will drop from fp32 to bf16 and reduce the batch size, but wanted to see if I could get any insight. Thanks for the help!

[Davido111200]

I didn't unfortunately. It seems like you gotta try multi-gpu training for this one

[Hprairie]

Ahh no worries, I appreciate the response!