Suggestions regarding the ExamPrepAgent

[ItshMoh]

In this issue comment thread, we will be discussing the features and bugs of ExamPrepAgent.
We can discuss what features we want and how to proceed with it.
We will also be sharing the bugs that we found upon testing the Agent.

[ItshMoh]

Hey @juntao
currently I am testing the BOT. While testing i found that LLama-nexus is not consistent with the tools use. It sometimes prints the tool_call. Even a small change in the system prompt llama-nexus response become inconsistent.
I am trying a lot of prompts which can be of our best use.

[juntao]

Yeah. Open-source models are not great at instruction following. So they are sensitive to prompts. We need to optimize both the system prompt and the mcp's tool description.

[ItshMoh]

Hey @juntao Have you tried the BOT?

[ItshMoh]

The Model responses are very inconsistent. For the same prompt it sometimes work properly and sometime not.

[ItshMoh]

The get_random_question this tool is very inconsistent. The description about this tool should be set very carefully

[juntao]

I think you can also put into the system_prompt in addition to tool description:

If the user asks for a new question or requests new mock exam, please use the get_random_question tool call to get the question and then respond to the user.

[juntao]

You can also try OpenAI or OpenRouter models in llama-nexus to see if their tool call capabilities are better. (You will need to update your llama-nexus binary).

curl --location 'http://localhost:3389/admin/servers/register' \
--header 'Content-Type: application/json' \
--data '{
    "url": "http://localhost:10010",
    "kind": "chat",
    "api_key": "Bearer <your-api-key>"
}'