clean up and add ckpt tests #179
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
samsja
force-pushed
the
refactor-test-and-hash
branch
3 times, most recently
from
0fa14d0 to
ca3b1c8
Compare
samsja
force-pushed
the
refactor-test-and-hash
branch
5 times, most recently
from
65e7b74 to
6755478
Compare
samsja
force-pushed
the
refactor-test-and-hash
branch
from
6755478 to
dbddd9f
Compare
samsja
changed the title
Jackmin801
requested changes
Dec 19, 2024
| memory_profiler: MemoryProfilerConfig | None = None | ||
|
|
||
| sequence_packing: bool = True | ||
| attn_fn: Literal["flash", "sdpa"] | None = None | ||
|
|
||
| math_attn: bool = False # slow |
There was a problem hiding this comment.
What do you think about putting this as an option in attn_fn instead?
There was a problem hiding this comment.
attn_fn: Literal["flash", "sdpa"] | None = None
@model_validator(mode="after")
def validate_attn_fn(self):
if self.attn_fn is not None:
warnings.warn("attn_fn argument is deprecated")
return self
hmm attn_fn is not used anymore. I just kept it to avoid conflict with old code.
There was a problem hiding this comment.
pr to remove attn_fn #180
| @@ -164,6 +200,7 @@ def train(config: Config): | |||
| config.type_model, | |||
| vocab_size=len(tokenizer) if config.name_model != "debugmodel" or not config.data.fake else TEST_VOCAB_SIZE, | |||
| seq_length=config.data.seq_length, | |||
| math_attn=config.train.math_attn, | |||
There was a problem hiding this comment.
What do you think about passing attn_fn instead? Would also allow sdpa to be specified
the profiler is enough I think I would kill it unless you had use case where you need it. I personally never used it even tho I added it haha |
|
yea never used it either haha |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
what this pr do: