Prompty, openai-messages-token-helper and tool calls

[gaborvar]

Hi,
Prompty apparently has some support for tool calls. It appears that prompty.prepare(prompt, data) compacts the calling ('assistant') message and the subsequent return ('tool') message into a single assistant message:

{'role': 'assistant', 'content': 'None\n\ntool:\n{"functionname": true, "result": ["...", "..."]}'}

The additional assistant message breaks the normal user/assistant cadence that promptmanager relies on:

azure-search-openai-demo/app/backend/approaches/promptmanager.py

Line 63 in d277675

for user_message, assistant_message in zip(remaining_messages[0::2], remaining_messages[1::2]):

Before considering a fix it would be useful to understand the future evolution of the message management part of the project. In addition to its token accounting feature, openai-messages-token-helper has overlapping features with the recently introduced prompty as both repackage messages (e.g. separating and adding back few-shots and latest user message etc.). I understand that tool calls are not prioritized in this project, however the growing code base translates into more maintenance to ensure that tool calls pass through transparently.

The best course of action also depends on prompty's stance towards tool_calls. Will the current support evolve ?

If tool_calls coexistence remains out of scope in this project then I welcome hints where I should look for a comparable scheme which is more friendly for this project. My current code builds tool calls dynamically to intelligently interface with the procedural world. Would structured outputs with response_format fit this project's philosophy better than tool_calls?

Thanks for the great work on the project

[pamelafox]

We definitely need to support either function calling, structured outputs, or function calling with structured outputs in this repo. I didn't realize there was currently an issue with the integration, looks like we need additional testing if I missed that.

As for prompty versus token-helper: The main goal of token-helper is to truncate conversation history to fit inside context windows. As the newer models have long context window lengths (128K), I'm beginning to think we may not need that truncation. It'd be great if we could remove the token-helper overhead and just assume the conversation fits inside the context window.
What are your thoughts on that?

As for structured outputs vs function calling: the main reason I haven't moved to structured outputs is that it's still not as well supported as function calling across the models and ecosystem. However, it is fairly well supported by Azure OpenAI models, so I'd be up for moving to that instead. Also curious what others think.

[gaborvar]

Hi,
For some background on making token-helper coexist with tool_calls you can look at these pull requests:

pamelafox/openai-messages-token-helper#23 (comment)
pamelafox/openai-messages-token-helper#21 (comment)
(this is not complete. I have made a few other updates recently - let me know if you are interested to adopt)

Goal of these PRs is not to support tool_calls with token counting but to transparently coexist with them. I did not attempt to count tokens of tool_calls and related messages but marked in the PR where that needs to be done.

But for longer term, I vote against fixing and maintaining token-helper. Although my project may upsize, currently I am far from depleting the context window. I only monitor token use and do rudimentary reaction if at all needed rather than the sophisticated ex-ante approach of token-helper.
If I have to choose, I choose tool-calls, not token counting.
Token-helper is also sensitive to typing. If I allow return values from the API to pass through to token-helper then it raises an exception. This should also be updated.
When OpenAI added the "audio" support to its API it also broke token-helper if used with tool_calls.

What do others think?