Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added sorting option for sentences during encoding #45

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+11 −3
Open

Added sorting option for sentences during encoding #45

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions sonar/inference_pipelines/text.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,7 @@ def predict(
max_seq_len: Optional[int] = None,
progress_bar: bool = False,
target_device: Optional[Device] = None,
sort_sent: bool = True,
) -> torch.Tensor:
"""
Transform the input texts (from a list of strings or from a text file) into a matrix of their embeddings.
Expand All @@ -165,12 +166,19 @@ def truncate(x: torch.Tensor) -> torch.Tensor:
nonlocal n_truncated
n_truncated += 1
return x[:max_seq_len]

def sort_input(input_sent) -> Sequence[str]:
return sorted(input_sent, key=len) if sort_sent else input_sent

if isinstance(input, (str, Path)):
input_data = read_sequence(sort_input(read_text(input)))
input_data = read_text(input).and_return()
else:
input_data = read_sequence(sort_input(input))

pipeline: Iterable = (
(
read_text(input)
if isinstance(input, (str, Path))
else read_sequence(input)
input_data
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at the end we want to return vectors in the same order as their original texts, so we need to memorize argsort permutation and to apply the inverse one before returning the result.

)
.map(tokenizer_encoder)
.map(truncate)
Expand Down
Loading