Use new prompt for QuestionAnswerTool

[cecheta]

Required by #322 , closes #648

Purpose

• This PR includes changes the prompt used in the QuestionAnswerTool to the main_prompt used in On Your Data.
• The new prompt includes a mix of system, user and AI messages, instead of one single user message
• The new prompt also includes an example to help the LLM, which can be configured on or off
• For backwards compatibility, if the old prompt has been set in the config, then this will continue to be used, with no changes
• Increase unit test coverage (not for streamlit)

The new prompt + few-shot example are configurable in the Admin app. There is JSON schema validation on the retrieved documents to warn against uploading invalid JSON.

Does this introduce a breaking change?

[ ] Yes
[x] No

Pull Request Type

What kind of change does this Pull Request introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

How to Test

• Run app and admin app locally
• Ask questions from the frontend
• Change the prompt in the Admin app

What to Check

• Check new prompt is being used
• Configure prompt in Admin app
• Configure few-shot example

[github-actions]

Coverage Report •

File	Stmts	Miss	Cover	Missing
code
app.py	14	14	0%	1–4, 6–7, 9, 12–14, 16, 18, 20–21
create_app.py	148	3	97%	199, 204, 327
code/backend
Admin.py	23	23	0%	1–6, 9, 11, 13–14, 16, 19–20, 22–23, 26, 33, 40, 43–45, 47, 49
code/backend/batch
function_app.py	16	16	0%	1–8, 10, 12–13, 15, 18–21
code/backend/batch/utilities/helpers
ConfigHelper.py	79	1	98%	113
EnvHelper.py	120	9	92%	194, 199–200, 203–205, 215–217
code/backend/batch/utilities/tools
QuestionAnswerTool.py	65	2	96%	44–45
code/backend/pages
04_Configuration.py	118	118	0%	1–8, 10, 12, 14, 21, 28, 30, 35–40, 43–48, 51–52, 55–66, 68–69, 79–81, 83–86, 89–91, 94–95, 100–101, 105–107, 110–111, 114–115, 118–119, 142, 144–145, 147–151, 153–156, 159–163, 170–171, 174–177, 179, 199–200, 202, 204, 210, 217, 224, 232, 239–240, 247, 249–250, 254, 261, 266, 272, 284–285, 302–303, 307, 309–310, 326, 358–359, 361–362
TOTAL	1827	830	54%

Tests	Skipped	Failures	Errors	Time
89	0 💤	0 ❌	0 🔥	10.469s ⏱️

[cecheta]

Get review from Project Wednesday before merging

[cecheta]

Putting in draft for now, to attempt to add few-shot example to the config Added to config

[adamdougal]

This is great! Only a couple of minor comments!

(I'll hold off approving until the review next week)

[superhindupur]

I understand that the prompt has been copied from Azure OpenAI On Your Data, but this is a very large prompt, and we've been told by data scientists in the past that long prompts don't work very well since the LLM tends to only "remember" the last few sentences and forget what was said to it in the beginning (attention bias, I think it is called). Should we get this reviewed by a data scientist?

[superhindupur]

My concern is that we don't have a standard process for DS evaluation - what if we make the performance worse.

[cecheta]

I agree that we should get as many reviews as possible, do you know who could review this for us?

[superhindupur]

We could check with Malvina/Eran if they can take a look at this. It doesn't have to be a blocker for this PR though, since users are allowed to change the prompt if they want to. Can totally be done as a follow-up.

[cecheta]

I am planning to add tests to cover these scenarios once the core of the PR has been reviewed

[cecheta]

Will create new pull request, due to vast number of changes since opening