Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tensor.item should be disallowed in the CUDA Graphs region #727

Open
IvanYashchuk opened this issue Jul 8, 2024 · 1 comment
Open

Tensor.item should be disallowed in the CUDA Graphs region #727

IvanYashchuk opened this issue Jul 8, 2024 · 1 comment
Assignees

Comments

@IvanYashchuk
Copy link
Collaborator

Note: If you have a model or program that is not supported yet but should be, please use the program coverage template.

🐛 Bug

import thunder
import torch

def func(x):
    return x.sum().item()

a = torch.randn(3, 3, device="cuda")

jfunc = thunder.jit(func)
jfunc(a) # works!

jfunc = thunder.jit(func, use_cudagraphs=True)
jfunc(a) # fails!
File thunder.CUDAGraph0_fn_3:10, in CUDAGraph0_fn(***failed resolving arguments***)
      9 del x
---> 10 f1 = Tensor.item(t0)  # f1: "float ?"
     11 del t0

RuntimeError: CUDA error: operation not permitted when stream is capturing
@tfogal
Copy link
Collaborator

tfogal commented Jul 8, 2024

related to #728

@nikitaved nikitaved self-assigned this Jul 8, 2024
@t-vi t-vi assigned t-vi and unassigned nikitaved Aug 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants