Curious about the torch.compile baseline in benchmark_litgpt

[mitkotak]

Was curious to know why the reduce-overhead option was not used as a baseline and if there's a comparison somewhere using reduce-overhead ?

Thanks !

cc @apaz-cli

[carmocca]

It should be because we are also not enabling CUDAGraphs with Thunder too ("reduce-overhead" simply enables CUDAGraphs). So this way is more of an apples-to-apples comparison. Maybe @parthmannan can correct me here

Either way, from my personal experience in running this, cudagraphs won't help during training (especially if your model is large enough) because you are not overhead bound at all

[mitkotak]

Thanks for the reply ! How about max-autotune-no-cudagraphs ?

[carmocca]

Sure, it could be used. Anecdotically, I found it to provide tiny speedups (if any) for a whole lot of compilation time, but it's a valid suggestion.