Skip to content

Commit 13fa50d

Browse files
committed
fix: Pad only depending on the largest sequence
1 parent 967dcd2 commit 13fa50d

File tree

1 file changed

+1
-2
lines changed

1 file changed

+1
-2
lines changed

include/models/basemodel.py

+1-2
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,7 @@ def tokenize(self, texts):
2626
models_logger.debug(f"Tokenising {len(texts)} texts")
2727
return self.tokenizer(
2828
texts,
29-
max_length=self.max_length,
30-
padding="max_length",
29+
padding="longest",
3130
truncation=True,
3231
return_tensors="pt"
3332
)

0 commit comments

Comments
 (0)