Remove requires_grad from regular TensorProxy and add specialized TensorProxy with requires_grad and is_leaf attributes #1599
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mruberry
reviewed
Dec 30, 2024
| @@ -1868,6 +1844,52 @@ def real(self): | |||
| return method(self) | |||
|
|
|||
|
|
|||
| class RuntimeTensorProxy(TensorProxy): | |||
| def __init__( | |||
There was a problem hiding this comment.
Would you add a docstring explaining this class's use vs. TensorProxy?
mruberry
reviewed
Dec 30, 2024
|
I think the idea of a Separately, how challenging would it be to fix the existing meta functions to propagate the requires_grad and is_leaf attributes correctly? Is that something we should start working on? |
IvanYashchuk
commented
Dec 31, 2024
thunder/core/prims.py
Outdated
There was a problem hiding this comment.
Changes in this file should be submitted in a separate PR.
… test_auto_register_torchops.py tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PyTorch tensors have attributes that affect the computation and Thunder should mirror those in TensorProxies. However, some of the attributes (
requires_grad,is_leaf,stride) are tricky to model precisely and propagate the values in Thunder's meta functions. This PR removes the possibility of querying and specifyingrequires_gradon regular TensorProxies which are used for intermediate values and reintroduces the attribute with a special classRuntimeTensorProxythat should be constructed through a corresponding real PyTorch tensor.A new attribute is added,
is_leaf, which is useful for raising errors when in-place copies should be disallowed (#1577, #1458).It's not part of this PR but an extension to query static strides information will be easy to add in the future.
Ref #1577, #1570.