Sunset thunder/benchmarks/distributed.py
and Improve thunder/benchmarks/benchmark_litgpt.py
#158
Labels
thunder/benchmarks/distributed.py
and Improve thunder/benchmarks/benchmark_litgpt.py
#158
https://github.com/Lightning-AI/lightning-thunder/blob/cdd43a7fc1110eec10f1854250299b84d1c3b2a8/thunder/benchmarks/distributed.py has been useful but I would find it not easy to extend, e.g. to support gradient accumulation.
https://github.com/Lightning-AI/lightning-thunder/blob/cdd43a7fc1110eec10f1854250299b84d1c3b2a8/thunder/benchmarks/benchmark_litgpt.py would be easy to work with as in #45 which is adding gradient accumulation with
no_sync
.cc @crcrpar @carmocca @awaelchli
The text was updated successfully, but these errors were encountered: