Use open_llama_7b_v2 for NER tasks. The result is problematic

[HyramXie]

I wrote the code according to the documentation, but there is a problem with the result

[rmitsch]

Hi @Hongxy-prodigal! Please attach code always as formatted text and not as picture.

• Which version of spacy-llm are you using?
• Please include the content of your config.cfg file.

[HyramXie]

My spacy-llm version number is 0.6.2

The main code:

from spacy_llm.util import assemble

examples_path = None
nlp = assemble(
    "config.cfg",
    overrides={}
    if examples_path is None
    else {"paths.examples": str(examples_path)},
    
)


text = """
Developers are not revealing the exact height of the Burj Dubai until its
official debut.  But they say the $1.5 billion tower will reach 200 stories
into the sky.

It will stretch 300 meters higher than Taiwan's Taipei 101, which is currently
holds the world record at 508 meters.

Dubai leaders hope the tapered metal and glass skyscraper will help restore
investors' confidence in the emirate.
"""

doc = nlp(text)
print([(ent.text, ent.label_) for ent in doc.ents])

The config.cfg:

[nlp]
lang = "en"
pipeline = ["llm"]
batch_size = 128

[components]

[components.llm]
factory = "llm"

[components.llm.model]
@llm_models = "spacy.OpenLLaMA.v1"
name = "open_llama_7b_v2"

[components.llm.task]
@llm_tasks = "spacy.NER.v2"
labels = DATE,GPE,LOCATION,NORP,ORGANIZATION,PERSON,TIME,EVENT,PRODUCT,WORK_OF_ART,FACILITIES
examples = null

[components.llm.task.normalizer]
@misc = "spacy.LowercaseNormalizer.v1"

[rmitsch]

Thanks for the config & script, both look fine. The issue in this case is the capability of the model to reply correctly to the prompt. Older (such as OpenLLamA) and smaller models sometimes have issues replying in the requested format, which breaks the parsing.
There are multiple ways in which you can improve your results:

• Use a newer/larger model.
• Add descriptions to your labels.
• Use spacy.NER.v3 instead of spacy.NER.v2. .v3 uses a chain-of-thought prompt and usually yields better results.

[HyramXie]

Thanks for your help