Givens orthogonal layer #57
Conversation
…ch-plugin-private into givens-rotation
…ch-plugin-private into givens-rotation
…ch-plugin-private into givens-rotation
|
I somehow broke @kevinchern's tests, what the hell... |
| def test_store_config(self): | ||
| with self.subTest("Simple case"): | ||
|
|
||
| class MyModel(torch.nn.Module): |
There was a problem hiding this comment.
Remove formatting changes. Is this "black" formatting?
There was a problem hiding this comment.
Yes. I have it by default on my vscode
@VolodyaCO which tests? I'm seeing |
…ch-plugin-private into givens-rotation
…plugin into givens-rotation
…plugin into givens-rotation
…plugin into givens-rotation
I forgot to update my tests to float64 precision. Now that I've done it, it's weird that all of the current failing tests are failing on File "/Users/distiller/project/tests/test_nn.py", line 144, in test_LinearBlock
self.assertTrue(model_probably_good(model, (din,), (dout,))) |
Ahhhhhh. OK Theo also flagged this at #50 . It's a poorly-written test.. you can ignore it. |
…plugin into givens-rotation
…ch-plugin-private into givens-rotation
2 participants
This PR adds an orthogonal layer given by Givens rotations, using the parallel algorithm described by Firas in https://arxiv.org/abs/2106.00003, which gives a forward complexity of O(n) and backward complexity of O(n log(n)), even though there are O(n^2) rotations.
This PR still is in draft. I wrote it for even n. Probably some more unit tests are to be done, but I am quite lazy (will do it after all math is checked for odd n).