Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for more complex parallelism strategies to benchmark_litgpt.py #77

Merged
merged 15 commits into from
Mar 27, 2024
Merged

Add support for more complex parallelism strategies to benchmark_litgpt.py #77

merged 15 commits into from
Mar 27, 2024

Conversation

parthmannan
Copy link
Collaborator

@parthmannan parthmannan commented Mar 25, 2024 •
edited
Loading

What does this PR do?

Adds support for more complex parallelism strategies -

  1. Hybrid FSDP using device mesh for PyTorch Eager/Torch.Compile
  2. Bucketing for FSDP based on size for PyTorch Eager/Torch.Compile
  3. Arg for bucketing for DDP based on size

cc - @carmocca @IvanYashchuk

parthmannan and others added 12 commits March 25, 2024 21:19
@parthmannan
Adding parallelism strategies to benchmark_litgpt.py
0980d2c
@parthmannan
Merge branch 'main' of https://github.com/Lightning-AI/lightning-thunder
57af94a 
 into pmannan_benchmark_litgpt
@parthmannan
Fixes to Hybrid FSDP and assert
c6d4074
@parthmannan
Add sharding_size assumption check
bc4dec9
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
32aaa91 
for more information, see https://pre-commit.ci
@parthmannan
Print the additional configurations when displaying results
e152da3
@parthmannan
Merge branch 'pmannan_benchmark_litgpt' of https://github.com/Lightni…
b350690 
…ng-AI/lightning-thunder into pmannan_benchmark_litgpt
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
ae5bb3e 
for more information, see https://pre-commit.ci
@parthmannan
Fix print mistake
b8b4aa9
@parthmannan
Merge branch 'pmannan_benchmark_litgpt' of https://github.com/Lightni…
0e9949e 
…ng-AI/lightning-thunder into pmannan_benchmark_litgpt
@parthmannan
Add support for _HYBRID_SHARD_ZERO2 for Eager/Torch.Compile
f303ec8
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
7bd677a 
for more information, see https://pre-commit.ci
@parthmannan parthmannan marked this pull request as ready for review March 27, 2024 01:23
@parthmannan
Copy link
Collaborator Author

@carmocca Ready for review! 🙌

carmocca
carmocca approved these changes Mar 27, 2024
View reviewed changes
Copy link
Contributor

@carmocca carmocca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

thunder/benchmarks/benchmark_litgpt.py Outdated Show resolved Hide resolved
thunder/benchmarks/benchmark_litgpt.py Outdated Show resolved Hide resolved
thunder/benchmarks/benchmark_litgpt.py Outdated Show resolved Hide resolved
thunder/benchmarks/benchmark_litgpt.py Outdated Show resolved Hide resolved
@carmocca carmocca enabled auto-merge (squash) March 27, 2024 01:42
parthmannan and others added 3 commits March 27, 2024 12:49
@carmocca carmocca merged commit 716fffd into main Mar 27, 2024
35 of 37 checks passed
@carmocca carmocca deleted the pmannan_benchmark_litgpt branch March 27, 2024 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@carmocca carmocca carmocca approved these changes

@mruberry mruberry Awaiting requested review from mruberry mruberry is a code owner

@lantiga lantiga Awaiting requested review from lantiga lantiga is a code owner

@robieta robieta Awaiting requested review from robieta

@t-vi t-vi Awaiting requested review from t-vi t-vi is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@parthmannan @carmocca