Add support for response format in tools #74

OskarStark · 2024-10-01T06:45:56Z

closes #49

I avoided to add logic for closures for now

src/StructuredOutput/ChainProcessor.php

src/ToolBox/AsTool.php

OskarStark · 2024-10-01T09:31:26Z

src/ToolBox/ChainProcessor.php

@@ -41,6 +41,8 @@ public function processOutput(Output $output): mixed
                $messages[] = Message::ofToolCall($toolCall, $result);
            }

+            // we need to check if the tool has a response format defined and apply that to the following call


TODO, but before doing it, I wanted to make sure the current state off ToolBox\ChainProcessor works as expected

OskarStark · 2024-10-01T09:32:40Z

src/ToolBox/Metadata.php

@@ -11,13 +11,15 @@
 {
    /**
     * @param ParameterDefinition|null $parameters
+     * @param array<mixed>|null        $responseFormat


The resolving of the responseFormat should be done before, so that we just have to deal with null (no response format) or a valid response format (array) here in the Metadata class, I hope you agree on that

src/ToolBox/Metadata.php

src/ToolBox/ToolAnalyzer.php

tests/ToolBox/ChainProcessorTest.php

OskarStark · 2024-10-01T09:55:24Z

tests/ToolBox/ToolAnalyzerTest.php

reworked that test a bit and added two more tests for response format

src/ToolBox/Attribute/AsTool.php

OskarStark · 2024-10-04T09:50:36Z

Rebased

chr-hertel · 2024-10-04T09:51:27Z

does it make sense to bring in an example here?

OskarStark · 2024-10-04T10:24:53Z

Yes, 👍🏻

OskarStark · 2024-10-04T20:36:20Z

examples/toolbox-clock.php

@@ -33,4 +33,5 @@
 $messages = new MessageBag(Message::ofUser('What date and time is it?'));
 $response = $chain->call($messages);

-echo $response.PHP_EOL;
+// Clock tool forced a structured output
+dump(json_decode($response, true));


This is now working @chr-hertel, but maybe you can help me polish the part in ChainProcessor. I am also not that deep in the ChainProcessor thing and I really don't get the difference between response_format and output_structure there. Thanks

chr-hertel · 2024-10-04T23:20:47Z

src/ToolBox/ChainProcessor.php

            foreach ($response->getToolCalls() as $toolCall) {
+                foreach ($this->toolBox->getMap() as $metadata) {
+                    if ($metadata->name === $toolCall->name
+                        && null !== $metadata->responseFormat
+                    ) {
+                        $options = array_merge($options, [
+                            'response_format' => [
+                                'type' => 'json_schema',
+                                'json_schema' => [
+                                    'schema' => $metadata->responseFormat,
+                                    'name' => $metadata->name,
+                                ],
+                            ],
+                        ]);
+                        break;
+                    }
+                }
+
                $result = $this->toolBox->execute($toolCall);
                $messages[] = Message::ofToolCall($toolCall, $result);
            }

-            $response = $output->llm->call($messages, $output->options);
+            $response = $output->llm->call($messages, $options);


this cannot work like this. or maybe it does with the example, but

a) multiple tool calls will interfere with each other
b) the original output format get overwritten

just realized why my assumption was wrong. the processor calls the llm not the chain - this is what makes it hard.

chr-hertel · 2024-10-05T00:56:31Z

please have a look at

refactor: ResponseInterface as Chain return value #117

and

feat: introduce chain awareness in chain processors to nest them #118

if they provide what you were looking for

chr-hertel · 2024-10-05T19:29:38Z

not sure this is still needed after #118 - guess the format is tackled and additional messages or option changes should be possible with custom processor - or do i miss something?

OskarStark · 2024-10-05T20:24:04Z

not sure this is still needed after #118 - guess the format is tackled and additional messages or option changes should be possible with custom processor - or do i miss something?

What do you mean by "is not needed"? The change in the chain processor?

Feel free to overtake the PR, I "steh ein bisschen auf dem Schlauch" 😄

Extra messages can be ignored for now, but I think yes, they are doable by your other PR, showcasing the TestProcessor

examples/toolbox-clock.php

OskarStark · 2024-10-08T20:47:56Z

So the current state of my PR is still working after a rebase and provides me with the correct result. However, after removing my logic from the ChainProcessor it doesn't work anymore.

I would really appreciate if you could take over this PR, as I am bit lost here 😄 Thanks! 🙏

chr-hertel · 2024-10-09T13:36:28Z

Let's have a call maybe friday or sth when I'm back home

DZunke · 2024-11-14T11:07:10Z

Hey, did your call result in something? Just out if curiousity because otherwise i would maybe have an idea to try out because i have also Customizing of it in progres.

I could think of something like with the other processors.

OskarStark · 2024-11-14T13:07:46Z

Hey, my use case is:

In my function I call an API and I get an array with products like:

[
  {
    "name": "foo",
    "url": "/foo",    
  },
  {
    "name": "bar",
    "url": "/bar",    
  }
]

but I want to have dedicated response format, so what I do is, I just give it back as the response to the function call and in the same time I force a response format:

Demo code (this is not how a json schema should look like 😄):

{
  "product_name": string,
  "target_url": string,    
}

So I don't want to do the mapping myself, but the LLM should do it for me.
Again, but only if the answer is from THIS function.

I hope it is clear now

chr-hertel · 2024-11-15T17:11:46Z

@OskarStark The issue is that function calls are handled as collection. there can be multiple of them with one model response and this makes it hard to "understand" which response_format we should use, see https://github.com/php-llm/llm-chain/blob/main/src/ToolBox/ChainProcessor.php#L50C1-L55C81

the alternative that i can see is some kind of decorator or processor that isolates the extraction that you want as extra feature.

OskarStark · 2024-11-15T19:05:10Z

Where you able to mange 2 function calls at once ?

DZunke · 2024-11-15T20:46:09Z

Ah ok. So kind of what i have done, as i want to customize tool settings like description for the function and description of method parameters at runtime from users application settings, is to create my own toolbox with the help of the toolbox interface and overwrite the Metadata that are responded by the getMap functionality. So the tool box itself is already customizable for own purposes to give back own interpretation of metadata objects.

What i take from your answer and code change is that you want to reach a changed tools mapping from metadata objects to array in some cases?

What my first simple idea was to extend the toolbox itself to build the mapping for the chain processors. This way it could also be possible to have totally different tool definitions beside the attributes that are mapped to metadata objects. This would maybe also shrink the need with all the schema definitions combined within the classes with the simple string behavor as the metadata objects could be left out as a contract between platform and toolbox.

I would understand this as a possibility to have a diverse set of custom interpreations available. Also to build complex json schema communication within the tools with own decisions based on the own use case instead of having the library to decide how to merge those complex JSON Schema together to a valid thingy.

Maybe it should also be possible to have and extended tool box interface with methods for "render map as information which tools are available" and "optionaly render map for tools execution" as Input and Output in ChainProcessor seems target to be different?

My use case is also currently working with the interface in place, as i mentioned, but i had those thoughts in general for an improvement after i had a revisit to this pull request.

What do you think? Maybe i am totally wrong as i am not utilizing the json schema approach and never tried the structured output in LLMs. Just wanted to bring this a bit further with sharing my thoughts.

chr-hertel · 2025-01-04T16:26:03Z

@OskarStark this one is obsolete, isn't it?

OskarStark self-assigned this Oct 1, 2024