Persist System Prompt

[ahmedsomaa]

I am working on a feature that require something similar to Custom ChatGPT where you can provide initial context + details once to the model and then ask whatever you want without needing to provide the initial context for each request.

For the feature, I am using the LLM to categorize the type of difference between two objects. So I provided the generateObject with two prompts:

• system: this provides the initial context + steps + examples for the model.
• prompt: this provides the testing data the model would apply the steps on.

When I checked the usage in the results, it seems that the system prompt is sent on every request which is not what we're looking for.

So is there a way to provide this context once for the model? This is done on the server-side, consider this like an endpoint

[brennanmceachran]

Each api call to LLMs is completely separate and distinct from each other - with no carry-over. So if you want the LLM to follow a set of instructions, you'll need those instructions on every call.

In practice, you'll always be sending a system prompt on every request with very few exceptions. That also means you'll be sending the entire message/conversation thread on every request.

If your concern is about sending a lot of data between your client and your server (not server and LLM) then you might find this article about message persistence helpful (just note that the server still sends the entire thread to the LLM afterwards)

[ahmedsomaa]

This is mainly done on the server not between the client and the server. Just getting some rows from a database table and then send them for some analysis and then persist the analysis results in a different table in the db.

[brennanmceachran]

Then, yes, you'll need to send the system prompt on every call to the LLM.

What is your main reason for not wanting this? Might be able to point you in other directions...

ie: If your concern is cost:

• Some providers (eg openai) cache tokens automatically, so the token cost of that system call might be reduced by ~50%.
• If you don't care about real-time, then things like the batch api from openai might be helpful. They take 24hrs to process but are 50% off all tokens prices.
• You could explore fine-tuning a model to significantly reduce system prompt instructions while keeping output quality higher.

[ahmedsomaa]

Mainly about the cost. I've been looking for the fine-tuned option but my main concern was the ability to call them using the SDK or not as we don't want to use any other SDK, and I found out that it's possible for some models like google here:

https://sdk.vercel.ai/providers/ai-sdk-providers/google-generative-ai

Thank you so much for you help.