Skip to content

Commit

Permalink
Update wording on FSDP
Browse files Browse the repository at this point in the history
  • Loading branch information
rasbt authored Mar 26, 2024
1 parent 9ed80c6 commit f15aa0a
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ As shown in the plot above, Thunder achieves a 40% speedup in training throughpu

## Multi-GPU performance

Thunder also supports distributed strategies such as DDP, FSDP, and DeepSpeed (ZeRO2 and ZeRO3) for training models on multiple GPUs. The following plot displays the normalized throughput measured for Llama 2 7B without FP8 mixed precision; support for FSDP is in progress.
Thunder also supports distributed strategies such as DDP and FSDP for training models on multiple GPUs. The following plot displays the normalized throughput measured for Llama 2 7B without FP8 mixed precision; support for FSDP is in progress.

<div align="center">
<img alt="Thunder" src="docs/source/_static/images/normalized_training_throughput_zero2.png" width="800px" style="max-width: 100%;">
Expand Down

0 comments on commit f15aa0a

Please sign in to comment.