Error in sample code for BERTopic

[JoeGermuska]

I was trying to run the sample code on the BERTopic page. I'll be honest that I don't really know what I'm doing

One error I ran into, which I was able to solve, is that it appears to need a different pipeline. The sample code has

nlp = spacy.load('en_core_web_md', exclude=['tagger', 'parser', 'ner', 'attribute_ruler', 'lemmatizer'])

which needed to be changed to en_core_web_trf

After that, I got an error, Can't find factory for 'curated_transformer' for language English, which led me to realize that I needed to install spacy-curated-transformers

Now, when topics, probs = topic_model.fit_transform(docs) is invoked, I get

AttributeError: 'DocTransformerOutput' object has no attribute 'tensors'

that's where I got stuck, and started looking at other approaches. But it seems like it would be good for sample code to be runnable as much as possible.

[alanbuxton]

@JoeGermuska I got something similar to work with the following:

topic_model = BERTopic(embedding_model=nlp.pipe)

[mowne67]

Thanks for the solution! This has worked for me!

[2513502304]

In fact, using embedding_model=nlp.pipe will use the default model instead of the corresponding spaCy model. Please read the source code of the select_backend function in bertopic\backend\_utils.py. It returns an instance of the SentenceTransformerBackend("sentence-transformers/all-MiniLM-L6-v2") model.

[2513502304]

By the way, the judgment condition of nlp.pipe in the source code is: str(type(embedding_model)). It returns a method string, so it won't match any valid model and will only return the default model.