Extract filter instructions from query

[PGijsbers]

Extend the pipeline to explicitly extract filter instructions from the natural language prompt.
For example:

"Give me small datasets that have data on flowers."

could translate to "Give flower datasets" and use filters "fewer than 10k rows" and "fewer than 10 features".
The first would continue to the RAG pipeline, and the second two filters would be applied on the returned results and shown directly in the web UI.

[PGijsbers]

@SubhadityaMukherjee and @Taniya-Das will both explore this work. Subha will focus more on post-hoc filtering, and Taniya on bringing the filters to a database level. Both have strengths and weaknesses, and we want to explore a little bit their effects in practice.

[SubhadityaMukherjee]

As an update, initial post-hoc filtering was added for some filters. UI and further improvements still to be done.

[PGijsbers]

@Taniya-Das has also created a flow where filters are extracted in a way that these filters may be applied at the database level. She will try to integrate it with the current tool.

[Taniya-Das]

The structuring-query script can be found in this branch - https://github.com/openml-labs/ai_search/tree/feature/structuring_query
A user query is decomposed into two parts - the search part + structured filter part.

Example -
A user query,
"query": "Find a mushroom dataset with less than 10k size"
would be structured as,
Search Query: mushroom dataset
Filter(s): comparator=<Comparator.LT: 'lt'> attribute='NumberOfInstances' value=10000

the structured filters can then be translated into the appropriate format.

[PGijsbers]

@Taniya-Das I think we have working prototypes now, so perhaps we should close this issue. Actually integrating this with the app is captured in other issues #27 #26 as is evaluation #15 #18 and further working on improving things #20