How to construct the query in a real-world scenario.

[geraldzm]

Hi, thanks for your work.

I'm building a template repository using Next.js with the AI SDK to explore StructRAG in a more real-world scenario.

I have one question:

How is the query supposed to be constructed?

In this repo, we build the query passed to the router, structurizer, and utilizer like this:

            query = data['prompt_template'].format(instruction=data['instruction'], question=data['question'], docs="......")

I understand this is based on Loong's benchmark setup, but in a real-world app, is it reasonable to just use query = question?

Note: I didn’t find anything about this in the paper. If I missed it, I’d appreciate it if you could point me to the relevant section.

Any insights would be appreciated.

[Li-Z-Q]

Hi, thanks for your attention.

The issue you raised is crucial for building real-world applications, and I’m very happy to discuss it with you.

As you mentioned, to adapt to Loong’s benchmark setup, the method I used in the paper to construct the query is:

query = data['prompt_template'].format(instruction=data['instruction'], question=data['question'], docs="......")  

where the data['question'] is something like:
"Please categorize the financial reports of the companies listed above by 'Earnings per Share (EPS)' into the following groups: high EPS (above $1.00), medium EPS (between $0.00 and $1.00), low EPS (between $-1.00 and $0.00), and negative EPS (below $-1.00). Place companies into the same collection for the same category and into different collections for different categories."

And the data['instruction'] is something like:
"We kindly ask you to review the financial statements of the companies provided above and answer the following questions based solely on the information you have seen. If the question involves content not found in the financial statements, you may ignore this part and only answer the other parts."

The reason for constructing the query this way is: To facilitate the subsequent router, structurizer, and utilizer processes, this query must specify two key aspects.

1. Task Scenario – Otherwise, StructRAG will struggle to select the most appropriate structure. For example, in the case above, the task scenario is explicitly stated as financial analysis, so the router can accurately determine that a table structure should be used.
2. Required Information – Otherwise, the structurizer will have difficulty performing effective information extraction and structured construction. For example, the case above directly specifies the need for 'Earnings per Share (EPS)' information, enabling the structurizer to efficiently extract the relevant details from the documents.

Therefore, for the query in the real-world app you mentioned, I believe it must also specify the task scenario and required information.

• On the frontend of app, some guiding text can be added to encourage users to emphasize these two aspects when formulating their queries.
• On the backend of app, some prompts can be prepended to the query to ensure the LLM pays close attention to these two aspects.

Additionally, since the router, structurizer, and utilizer are all LLM-driven, the textual format and linguistic style of the query can be quite flexible and don’t need to adhere to a fixed format.

I hope above content is helpful. If you have any further points to discuss, feel free to leave a comment anytime!