STWFSA memory use is not reduced by subject exclusions during inferencing (and training)

[mjsuhonos]

I'll look into this further, but wanted to flag it as an issue. I haven't measured training RAM yet, but will update with more details when I can.

Vocab A: 100K labels
Vocab B: 2.1M labels, divided into 2 skos:ConceptScheme : 100K (same as the 100K in Vocab A) and 2M

(exclude=*,include_scheme=test:vocab_A)

STWFSA inferencing w/ Vocab A: 4GB RAM
STWFSA inferencing w/ Vocab B: >50GB RAM

Same suggestion results from both models.

data/projects/stwfsa_A/stwfsa_predictor.zip: 362845163 bytes
data/projects/stwfsa_B/stwfsa_predictor.zip: 6183918280 bytes

It appears that STWFSA saves the entire (unexcluded) vocabulary after training, thus vastly increasing RAM usage for both training and inferencing. This severely limits the ability to use STWFSA for large vocabularies with exclusion rules.

[osma]

This makes sense, because subject exclusion is more of a pre- and post-processing feature than something backend themselves are aware of. Models are still given the full vocabulary (in the case of stwfsa and mllm, the full SKOS file/graph), but the excluded subjects are filtered from training data and if the model suggests them anyway, they are also dropped from the suggestions.

This was easiest to implement; the alternative would be much more intrusive. But I see that this is far from ideal in your setting.