Skip to content

Commit

Permalink
Need to cache tokenize annotators based on the segment properties as …
Browse files Browse the repository at this point in the history
…well
  • Loading branch information
AngledLuffa committed Nov 27, 2024
1 parent 9732f82 commit 6f6eb93
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions src/edu/stanford/nlp/util/PropertiesUtils.java
Original file line number Diff line number Diff line change
Expand Up @@ -429,8 +429,9 @@ public static String getSignature(String name, Properties properties, Property[]
public static String getSignature(String name, Properties properties) {
String[] prefixes = new String[]{(name != null && !name.isEmpty())? name + '.' : ""};
// TODO(gabor) This is a hack, as tokenize and ssplit depend on each other so heavily
if ("tokenize".equals(name) || "ssplit".equals(name)) {
prefixes = new String[]{"tokenize", "ssplit"};
// the tokenize annotator also uses segment properties to determine which model to use, etc
if ("tokenize".equals(name) || "ssplit".equals(name) || "segment".equals(name)) {
prefixes = new String[]{"tokenize", "ssplit", "segment"};
}
// TODO [chris 2017]: Another hack. Traditionally, we have called the cleanxml properties clean!
if ("clean".equals(name) || "cleanxml".equals(name)) {
Expand Down

0 comments on commit 6f6eb93

Please sign in to comment.