Upgrade nvFuser in CI

[tfogal]

🚀 Feature

Please upgrade nvFuser to 0.2.22 or later.

Motivation

We are hitting bugs that are fixed in more recent versions of nvFuser, and this is blocking #1340 at least.

Additional context

The command you'd want to use to do this will depend on which version of torch and CUDA you are using.

• CUDA 12.4, torch 2.4: python3 -m pip install --extra-index-url https://pypi.nvidia.com --pre nvfuser-cu124-torch24
• CUDA 12.1, torch 2.3: python3 -m pip install --extra-index-url https://pypi.nvidia.com --pre nvfuser-cu121-torch23
• CUDA 12.4 nightly torch: python3 -m pip install --extra-index-url https://pypi.nvidia.com --pre nvfuser-cu124

[tfogal]

@Borda guessing this goes to you; apologies if not, please let me know!

[t-vi]

I'll take care of it.

[t-vi]

Note that the NVIDIA PyPI does not seem to have recent packages built against stable PyTorch, so there is little we can do without updates to the packages (torch25 might be interesting, too)...

e.g. https://pypi.nvidia.com/nvfuser-cu121-torch24/

[t-vi]

@xwang233

[xwang233]

Sorry about the delay there. We decided to push nvfuser build against torch stable to pypi.org twice a month (1st and 15th).

Nvfuser build against torch nightly will still be pushed nightly to pypi.nvidia.com every day.

I'll trigger a build for nvfuser build against torch stable 2.5 now to unblock CI upgrades.

[xwang233]

Also, the nightly and stable builds now support (python 3.10, and 3.12) x (cuda 11.8, 12.1, and 12.4)

[jjsjann123]

Nvfuser build against torch nightly will still be pushed nightly to pypi.nvidia.com every day.

do you mind updating the installation instruction in TomF's comment at the top? (for build against stable torch from pypi.org)

[t-vi]

We currently take nvfuser from PyPI, but I don't seem to see recent updates, e.g in https://pypi.org/project/nvfuser-cu121-torch24/#history

[xwang233]

We currently take nvfuser from PyPI, but I don't seem to see recent updates, e.g in https://pypi.org/project/nvfuser-cu121-torch24/#history

Builds against torch stable are ongoing and will be pushed to pypi.org in a few hours. Starting 11/1, builds against torch stable will be pushed twice a month to avoid delays in nvfuser features.

[tfogal]

... to avoid delays in nvfuser features.

Thank you, Xiao!

[xwang233]

nvfuser builds against torch25 are uploaded to pypi.org

[t-vi]

@xwang233 Thank you. I'll bump the CI to pt 2.5.1. Do you have plans for building torch24, too? If not we need to bump the docs to use 2.5 as well.

[xwang233]

Do you have plans for building torch24, too?

The automatic CI builds against stable is always for the latest torch stable release (pip install torch). If you need builds against old torch releases, we may be able to add those.