Skip to content

Commit

Permalink
fix segfault
Browse files Browse the repository at this point in the history
  • Loading branch information
pavel-esir committed Aug 26, 2024
1 parent a3fa982 commit 49b6e8a
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion src/cpp/src/tokenizer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,8 @@ class Tokenizer::TokenizerImpl {
// Initialize tokenizer's cache to save time later.
// infer_special_tokens_if_necessary() already could do that
// but it didn't run decode() for sure.
decode(encode("").input_ids);
// TODO CVS-150630: Empty strings sporadically can fail, therefore use nonempty string for warmup.
decode(encode("non empty string").input_ids);
}

// load special tokens ids from config.json
Expand Down

0 comments on commit 49b6e8a

Please sign in to comment.