Add custom timeout configuration for LLM operations

[dsilahcilar]

Currently, framework does not support configuring timeouts for individual LLM calls. All LLM operations use the default timeout values from the underlying LLM client, which cannot be customised per request.

Add timeout option like this:
val llmOptions = LlmOptions(,
temperature = 0.5,
timeout = Duration.ofSeconds(60) // ❌ Not supported
)

Use Cases:

• Long-running analysis tasks that need extended timeouts
• Different timeout requirements for different LLM providers or models
• Preventing hung requests on unreliable network connections

[HyunSangHan]

Can I take a look at this issue?

[dsilahcilar]

please go ahead.

[HyunSangHan]

Hi, team! 👋

I’ve been trying to add a timeout configuration to LlmOptions so that each request can have its own timeout value. I attempted this by extending LlmOptions and injecting the timeout value, expecting that it would be applied when calling createChatClient(). However, I can’t find a proper way to set the timeout on the ChatClient.

embabel-agent/embabel-agent-api/src/main/kotlin/com/embabel/agent/spi/support/springai/ChatClientLlmOperations.kt

Lines 229 to 235 in 1878dac

	private fun createChatClient(
	llm: Llm,
	): ChatClient {
	return ChatClient
	.builder(llm.model)
	.build()
	}

It doesn’t seem possible to add the timeout during the ChatClient building process, but I noticed that in the process of creating the Model object used by the ChatClient, it is possible to set a timeout inside the API object like this. : spring-projects/spring-ai#3573 (comment)

The issue is that in the current flow, the Model object is already built and finalized before building the ChatClient, so it’s too late to inject the timeout at that stage.

One approach I considered is using @Value to inject the timeout setting into the Model layer. However, this makes it a static configuration and makes it difficult to inject the timeout dynamically via LlmOptions.

Do you have any good ideas on how to achieve this? 🤔

[HyunSangHan]

By the way, I believe the best approach would be for the Spring AI project’s ChatClient to have built-in support for timeouts, so that we can simply make use of that functionality. That's why I create an issue like this: spring-projects/spring-ai#4103

[hadirsa]

Hi @HyunSangHan,

Looks like we were both working on this issue around the same time.
As you mentioned, SpringAI’s ChatClient currently doesn’t support timeouts. We can add this capability on the Embabel side for now, and once SpringAI introduces native support, we can adapt our implementation accordingly.

[HyunSangHan]

Hi @hadirsa,
I saw your PR. I agree that CompletableFuture is one of the simplest ways to address this issue, which was also my initial idea when I was first assigned to it last Friday.

However, I spent the past weekend exploring alternative approaches such as setting timeouts at the HTTP client level, because they can also terminate the server-side work and provide a more robust solution. (For reference, based on my research, it seems that Spring AI also recommends using HTTP client-level timeouts: spring-projects/spring-ai#3573 (comment))
This was the reason I started the discussion here: #664 (comment)

If the project prefers a more fundamental fix rather than just client-side timeout handling, I’d like to first discuss this direction with the maintainers in this issue. Since I’ve already put considerable effort into this and have much of the code ready, I’d be happy to finalize my approach and submit my code as PR afterward. :)

[hadirsa]

Hi @hadirsa, I saw your PR. I agree that CompletableFuture is one of the simplest ways to address this issue, which was also my initial idea when I was first assigned to it last Friday.

However, I spent the past weekend exploring alternative approaches such as setting timeouts at the HTTP client level, because they can also terminate the server-side work and provide a more robust solution. (For reference, based on my research, it seems that Spring AI also recommends using HTTP client-level timeouts: spring-projects/spring-ai#3573 (comment)) This was the reason I started the discussion here: #664 (comment)

If the project prefers a more fundamental fix rather than just client-side timeout handling, I’d like to first discuss this direction with the maintainers in this issue. Since I’ve already put considerable effort into this and have much of the code ready, I’d be happy to finalize my approach and submit my code as PR afterward. :)

Thanks @HyunSangHan for your thoughtful response and for sharing your approach! I appreciate the effort you’ve put into exploring the HTTP client-level timeouts and the context you provided from Spring AI’s recommendations.

I also came across a similar discussion here: spring-projects/spring-ai#354 (comment) which reinforces the idea that a more fundamental fix would be ideal.

If you’d like, you can also join the Embabel Discord to discuss this further. It might help us sync up more easily.

In the meantime, adding the timeout on the Embabel side seems like a practical step to handle this issue.

Looking forward to collaborating and finalizing a solid solution! Let me know if you’d like to sync up or if there’s anything I can do to help.

[dsilahcilar]

I think both efforts are equally valuable for this issue. Since you both agree that a more robust solution would be ideal, why don't we just start with @hadirsa's fix as an immediate fix then implement @HyunSangHan's Spring AI change once the code is ready? (I believe it may take quite a while until we have a release there)

@johnsonr what do you think?

[hadirsa]

I think both efforts are equally valuable for this issue. Since you both agree that a more robust solution would be ideal, why don't we just start with @hadirsa's fix as an immediate fix then implement @HyunSangHan's Spring AI change once the code is ready? (I believe it may take quite a while until we have a release there)

@johnsonr what do you think?

Thanks @dsilahcilar for the suggestion. I agree.

• We can start with timeout capability in Embabel and can improve it in the following days or phases.
• We can also add HTTP client-level timeout, while waiting for native ChatClient timeout capability from SpringAI.
• Or, if @HyunSangHan has already completed or is close to completing his changes, we can wait to review his implementation first.

I support the community's decision and will follow that direction.

[alexheifetz]

timeout feature is now available in embabel-agent-0.1.1-SNAPSHOT build. Thank you @hadirsa!
@dsilahcilar, @HyunSangHan please go ahead and test it out.