Models

[rbroc]

Looking both at foundation and instruction tuning models. For this project, the latter is probably going to be the only target, as it would probably work better.

Available

• Flan-T5: https://huggingface.co/google/flan-t5-xxl (on HuggingFace)
• Falcon: https://huggingface.co/tiiuae/falcon-40b (both base model and instruction tuned). Note that there is also a 7b instruction-tuned and a 7b version (on HuggingFace)
• Llama2 (probably just for comparison) and StableBeluga (https://huggingface.co/stabilityai/StableBeluga2)
• Alpaca (library, for both foundation model and instruction-tuned model)

Maybe for later
Not open-source

• GPT-4 (pricing 0.03$ / 1k tokens for prompts; 0.06 $ / 1k tokens completions) - (on hold, because instruction tuning version is not available)
• PaLM - (on hold)
• BARD
  Open-source
• Cerebras GPT: https://huggingface.co/cerebras/Cerebras-GPT-6.7B
• Blender for dialogue

[rbroc]

see #51: at the end of the whole process, we might want to:

• update this with the models we are actually using
• reconsider our current choices (e.g., is using LLaMaChat & Mistral Instruct fair? do we want to include more models?) on the basis of an updated picture of the LLM landscape.

This should be done at the end of the project though, not before - too many new models all the time!

[MinaAlmasi]

Models are sort of "out of date" by now, so we should probably consider new ones. ATM:

Updating LLMs (Mina's scribbles):

Llama3

• Apparently the 1b llama3 is better than llama2 chat 13b on some tasks?

Maybe stabilityai/stablelm-2-12b-chat ? (Since we are using stabilityai/beluga7b currently.).

• Whereas Stable Beluga 7b was a fine-tune of Llama. Stablelm seems to be a new model entirely
• Seems to perform worse than Gemma but better than llama2 and mistral 7b (see link for openLLM leaderboard)

We'll consult Kenneth when we are closer to having a polished pipeline

[rbroc]

Some input for this:

• We need to have a couple of versions of LlaMa2 and Llama3
• Some Mistral models
• Zephyr? https://huggingface.co/HuggingFaceH4/zephyr-7b-beta
• Some Qwen model
  We can define the exact versions right once we rerun the whole pipeline. @rdkm89, if you have input on any class of open-source models that should be included please do chime in.

[rdkm89]

Some input for this:

* We need to have a couple of versions of LlaMa2 and Llama3

* Some Mistral models

* Zephyr? https://huggingface.co/HuggingFaceH4/zephyr-7b-beta

* Some Qwen model
  We can define the exact versions right once we rerun the whole pipeline. @rdkm89, if you have input on any class of open-source models that should be included please do chime in.

I'm not sure that Llama 2 is relevant anymore, I'd probably go for at least 3.1 but preferably 3.2. Likewise, I think that Zephyr is a bit of a dead end.

I think my vote (right now, anyway) would be Llama 3.2, Mistral, Qwen 2, and Gemma 2.

[rbroc]

Awesome, let's run with that unless anything mindblowing is released in the meantime.