MLLM explodes on vocabularies with large number of skos:Collection members

[mjsuhonos]

Noticed this on Annif 1.4.1 but confirmed it on 1.3.1 as well.

I have a vocabulary with ~100K labels that are all part of a skos:Collection; ie. each entity has a skos:member statement. When using MLLM to train even a very small document set, eg. 5 documents, RAM usage during training rapidly spikes to the point that my laptop becomes unresponsive. On a VPS this gives a regular OOM condition (hundreds of GB).

When I remove only the skos:Collection and skos:member statements, MLLM training runs as normal.

[mjsuhonos]

After further investigation, this is a SKOS gotcha -- implementing set selection using skos:ConceptScheme doesn't exhibit this issue. I'm guessing rdflib doesn't like to do traversal with skos:Collection. It's not a bug, but definitely something to be aware of.

[osma]

Thanks for the report. But I'm a bit puzzled, could you explain this a bit more?

What is the pattern of collections vs. concepts in your vocabulary? I assume you meant that you have ~100k concepts. How many collections do you have? How many concepts per collection min/avg/max?

The way the collection feature works in MLLM is that it tries to figure out which concepts in the candidate set (potential matching subjects/concepts for a single sentence of document text) are part of the same SKOS collection. To do that, it constructs a sparse binary matrix of size (n_collections,n_concepts):

Annif/annif/lexical/util.py

Lines 44 to 58 in 27e4ac7

	def make_collection_matrix(graph: Graph, vocab: AnnifVocabulary) -> csc_matrix:
	# make an index with all collection members
	c_members = collections.defaultdict(list)
	for coll, member in graph.subject_objects(SKOS.member):
	member_id = vocab.subjects.by_uri(str(member), warnings=False)
	if member_id is not None:
	c_members[str(coll)].append(member_id)
	c_matrix = lil_matrix((len(c_members), len(vocab.subjects)), dtype=bool)
	# populate the matrix for collection -> subject_id
	for c_id, members in enumerate(c_members.values()):
	c_matrix[c_id, members] = True
	return csc_matrix(c_matrix)

Then when computing features for individual candidates, it uses this matrix to map concepts to other concepts. In practice this is done by computing (c_vec ⊙ M)ᵀ M, here:

Annif/annif/lexical/mllm.py

Line 119 in 27e4ac7

collection = mdata.collection.multiply(c_vec).T.dot(mdata.collection).sum(axis=0)

I think it's likely that somehow this is the part that explodes. You can try commenting out these parts of the code to see if it makes the problem go away.

[mjsuhonos]

Thanks for the explanation -- I'll poke at that line and see how it affects training.

My use case is a large unbalanced vocabulary made of two parts: A contains concepts (90K terms), B contains names (2M terms). I use the exclusion functionality to train each part individually, then combine them for suggesting. When generating them, I was creating a collection for each part; ie. A is a collection of 90K members, and B is a collection of 2M members. I realize now that this produces a large number of skos:member triples on the skos:Collection object. This is both semantically and technically wrong; using ConceptScheme moves the membership triples (using skos:inScheme) to each term object, which then works fine.

The take-away for me was that skos:Collection should only hold a smaller number of concepts, and isn't the right approach for this use pattern in Annif. But perhaps a guard condition for collections above a certain size to prevent catastrophic OOM would be a good safety measure.