Skip to content

Replacing uniform with uniform_philox , and incrementing PRNG state#237

Merged
t-vi merged 9 commits intorng_update_primsfrom
uniform_rng
May 29, 2024
Merged

Replacing uniform with uniform_philox , and incrementing PRNG state#237
t-vi merged 9 commits intorng_update_primsfrom
uniform_rng

Conversation

@kiya00
Copy link
Copy Markdown
Collaborator

@kiya00 kiya00 commented Apr 19, 2024

This PR is one step towards recomputing dropout in backward, the steps are:

Ref #114

@kiya00 kiya00 requested a review from IvanYashchuk April 19, 2024 10:47
@kiya00 kiya00 marked this pull request as draft April 19, 2024 10:48
@kiya00
Copy link
Copy Markdown
Collaborator Author

kiya00 commented Apr 22, 2024

TODO:
fix #231
update the offset according to nvfuser properly

@IvanYashchuk
Copy link
Copy Markdown
Collaborator

Let's split replace_uniform into a separate PR.
Primitives for getting and setting the state can be tested without replace_uniform and having #231 fixed is not required for these primitives.

@kiya00 kiya00 changed the base branch from main to rng_update_prims April 24, 2024 15:27
@kiya00 kiya00 marked this pull request as ready for review May 7, 2024 13:09
@kiya00 kiya00 force-pushed the rng_update_prims branch from f9ec34f to f054d57 Compare May 28, 2024 11:36
@kiya00
Copy link
Copy Markdown
Collaborator Author

kiya00 commented May 28, 2024

Hi @mruberry @jjsjann123 @IvanYashchuk , I modified it according our design review discussion, could you take a look?

Copy link
Copy Markdown
Collaborator

@mruberry mruberry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll let @IvanYashchuk do the final review. Thanks for the design review, @kiya00!

@t-vi t-vi merged commit 1a74d84 into rng_update_prims May 29, 2024
@t-vi t-vi deleted the uniform_rng branch May 29, 2024 15:45
@kiya00 kiya00 restored the uniform_rng branch May 29, 2024 15:57
@IvanYashchuk
Copy link
Copy Markdown
Collaborator

merged 9 commits into rng_update_prims from uniform_rng

We had a manual stack here 😃

@t-vi
Copy link
Copy Markdown
Collaborator

t-vi commented May 29, 2024

Yeah, I'll learn to read...

@kiya00
Copy link
Copy Markdown
Collaborator Author

kiya00 commented May 29, 2024

sorry for the trouble, I should get that draft status on

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add torch.nn.Dropout recomputation support during the backward pass to Thunder

4 participants