Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sunset thunder/benchmarks/distributed.py and Improve thunder/benchmarks/benchmark_litgpt.py #158

Open
1 task
crcrpar opened this issue Apr 11, 2024 · 0 comments

Comments

@crcrpar
Copy link
Collaborator

crcrpar commented Apr 11, 2024

  • [cosmetic] improve the format of JSON output of benchmark_litgpt.py

https://github.com/Lightning-AI/lightning-thunder/blob/cdd43a7fc1110eec10f1854250299b84d1c3b2a8/thunder/benchmarks/distributed.py has been useful but I would find it not easy to extend, e.g. to support gradient accumulation.

https://github.com/Lightning-AI/lightning-thunder/blob/cdd43a7fc1110eec10f1854250299b84d1c3b2a8/thunder/benchmarks/benchmark_litgpt.py would be easy to work with as in #45 which is adding gradient accumulation with no_sync.

cc @crcrpar @carmocca @awaelchli

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant