Templating messages to be sent to llms

[meain]

I was wondering if there there was some way I could template out a longer conversation than just system prompt + context. Currently if I want do do some refactor on regions(let say I add the directive as "flip the args"), the messages that get sent to llm is:

• system: You are a large language model and a careful programmer. Provide code and only code as output without any additional text, prompt or note.\n\nflip the args
• user: <.... code that needs to be changed ....>

From my testing I found that llms work much better when the system message/prompt is just high level role and the instruction is separate. For example the messages would look something like this:

• system: You are senior software engineer who helps with code refactors. I'll provide you some code and instructions on what change needs to happen. Respond with only code.
• user: <.... code that needs to be changed ....>
• assistant: What is the change that needs to happen
• user: flip the args

To avoid any confusion, the above set of messages is not created over time, but loaded from a template with the dynamic parts filled in. Let me know if I'm missing something, but as far as I can tell gptel does not allow to create such a list of messages to be sent to llms. It is even more useful when we have to add additional context to add some metadata around them.

---

I've been hacking around these concepts on meain/yap and the responses with messages structured this way seem to be much better even with smaller models. I build out messages based on the user selection and "directive"/user-prompt. Also see here to see how one can provide additional data and nudge the llm to only use it for context.

[meain]

Kinda related: #249

[karthink]

Neat idea! The TL;DR is that I'm on board with the idea, but not sure about the implementation.

For gptel I'm not interested in providing opinionated/fit-to-purpose system messages or templates for tasks, since (for the most part) I don't want to be in the business of testing various templates to see which ones work best, and changing them over time. gptel is 75% basic infrastructure and 25% UI.

However I am interested in providing the flexibility to create your own templates like the ones in yap, preferably with some UI. This is part of a larger plan to allow gptel to specify an LLM pipeline interactively or with elisp, i.e. a linked graph where each node is an LLM API call and you can specify the parameters for each, including templates like the ones from your experiments. You can already do this by chaining together gptel-request callbacks, but I've been trying to figure out a better declarative way to specify the tree.

The problem so far is that users are comfortable with the familiar chat and "operate on this region in place" paradigms, but it's not apparent to them how to use an LLM client to simulate any other kind of use. For example, you have to know elisp and read the documentation of gptel-request to figure out how to simulate your "summarize a url" template.

Other LLM clients (or web services) offer a plethora of specialized commands like -summarize out of the box, but I find this constraining and prescriptive, and I want to find a way to provide composable building blocks. Hope that makes sense.

I'm curious to know if you have any thoughts about this approach.

[meain]

For gptel I'm not interested in providing opinionated/fit-to-purpose system messages or templates for tasks, since (for the most part) I don't want to be in the business of testing various templates to see which ones work best, and changing them over time. gptel is 75% basic infrastructure and 25% UI.

I agree, I don't think gptel should be responsible for managing prompts. I mostly just wanted to include a few sample templates in yap just so that people have an idea of the kind of things they can build. I expect that in most cases, users have their specific templates. Another option would be to have a separate "contrib" repo with less oversight that can collect just these templates.

On that note, a lot of even my own personal usecase is just providing just the last user prompt for the yap-rewrite function. My workflow would be something like:

• M-x yap-rewrite
• Provide user prompt (something like "add emoji to docstring")
• Yap templates out the prompt based on the default template (shown below)
• Rewrites the code with one received from llm (with some additional UI niceties)

default template:

• system: You are blah, blah, blah. I'll give you some code and transformation....
• user:
• system: What is the transformation?
• user: <user prompt: "add emoji to docstring">

Just this alone would be huge as the current way of just adding the transformation in the system message produces worse results in my testing.

However I am interested in providing the flexibility to create your own templates like the ones in yap, preferably with some UI.

While being able to define templates via UI is good, I personally think for more most use-cases, we should have predefined templates. Most of the times, I just want to select a region and give it a user prompt and have the tool just template out a chat so as to optimize the llm output.

The problem so far is that users are comfortable with the familiar chat and "operate on this region in place" paradigms, but it's not apparent to them how to use an LLM client to simulate any other kind of use.

I think most end users only do have to rely on this. As I said, once you have a good enough generic(probably mode aware template), it is just about selecting a region and giving a command. The main idea here is that we should split out the transformation to the region and system prompt as separate messages. See my yap-rewrite workflow above.

Other LLM clients (or web services) offer a plethora of specialized commands like -summarize out of the box, but I find this constraining and prescriptive, and I want to find a way to provide composable building blocks.

I totally agree. Providing specialized commands is very restricting. Even in yap, I've avoided providing commands. There are a few builtin templates, but they are more of "these are the kind of things you can do". I have however focused a bit more on providing functions that can let you build templates like yap-template-selection-context and yap-template-buffer-context. Also see docs.

---

Here is a video of how I use yap-rewrite for additional context. If you see, I just provide a single "user prompt" after selecting the region.

Screen.Recording.2024-09-08.at.9.27.20.AM.mov

[meain]

I agree this is more of an "how to best design the API" question. What gptel does best for me is the ability to add context from different sources and use that for conversation. I would like to plug addition context providers into gptel transient dynamically, but that is a different discussion and issue to be filed. I can work with just writing functions with gptel-add to add additional context for now.

[karthink]

@meain I have added support for templates to gptel. It's currently in the feature-templates branch. If you get a chance, I'd appreciate your feedback on the API and on its performance.

The commit message doesn't explain much, so for now here's how you can use them:

gptel-directives is now a map from directive names to directives, where a directive is more general than a system message and does not have to be a string.

A directive can be

• A string, interpreted as the system message.
• A list of strings, whose first (possibly nil) element is interpreted as the system message, and the remaining elements as alternating user prompts and LLM responses. This can be used to template the intial part of a conversation.
• A function that returns a string or a list of strings, interpreted as the above. This can be used to dynamically generate a system message and/or conversation template based on the current context.

Here are some examples of directives:

A string

Interpreted as a simple system message. No change from gptel's current behavior:

"You are a tutor and domain expert in the domain of my questions.  You will lead me to discover the answer myself by providing hints.  Your instructions are as follows:
- If the question or notation is not clear to you, ask for clarifying details.
- At first your hints should be general and vague.
- If I fail to make progress, provide more explicit hints.
- Never provide the answer itself unless I explicitly ask you to.  If my answer is wrong, again provide only hints to correct it."

A list of strings

A template consisting of a system message followed by user/assistant interactions (synthetic example since I couldn't think of a static example that doesn't need a function):

("You are a large language model living in Emacs and a helpful assistant. Respond concisely." ; system
 "<user question for priming the llm here>"     ;user
 "<canned llm response here>")                  ;llm

A function that returns a string

A string returned by a function is interpreted as the system message. A forced example here, since these can be three different system messages that you can manually select.

(defun mode-specific-instruction ()
  "Return a generic LLM system prompt depending on context."
  (cond
   ;; In programming modes
   ((provided-mode-derived-p major-mode 'prog-mode)
    (let ((lang (gptel--strip-mode-suffix major-mode)))
      (format
       (concat "You are a %s programmer.  Answer my questions with a combination of %s code and text."
               "  The text should be laced in appropriate code comments.")
       lang lang)))
   
   ;; When composing git commits
   ((or git-commit-mode
        (buffer-match-p "^COMMIT_EDITMSG$" (current-buffer)))
    (concat "You are a git commit helper.  Generate a changelog for this changeset:\n\n```diff"
            (shell-command-to-string "git diff")
            "```"))

   ;; In text modes
   ((provided-mode-derived-p major-mode 'text-mode)
    (concat "You are a prose editor.  Proofread the provided text and suggest changes based on:\n"
            "- Grammar and style: check for fragments, run-on sentences, purple prose and the like."
            "- Brevity: Suggest ways to shorten or simplify this text without losing important details."))))

The first element of the list can be nil to enforce no system message.

A function that returns a list of strings

A function that returns a template, including the system message:

(defun rewrite-template ()
  "Rewrite or refactor selected region"
  (let ((lang (downcase (gptel--strip-mode-suffix major-mode))))
    (list ;; system
          (format (concat "You are a %s programmer.  "
                          "Follow my instructions and refactor %s code I provide.  "
                          "Generate ONLY %s code as output, without "
                          "any explanation or markdown code fences.")
                  lang lang lang)
          ;; user
          (buffer-substring-no-properties (region-beginning) (region-end))
          ;; llm
          "What is the required change?")))

Using these directives in gptel

These can be added to gptel-directives.

(setq gptel-directives
      `((tutor     . "You are a tutor and domain expert in the domain of my questions...")
        (synthetic . ("You are a large language model living in Emacs and a helpful assistant. Respond concisely."
                      "<user question for priming here>"
                      "<canned llm response here>"))
        (DTRT      . ,#'mode-specific-instruction)
        (rewrite   . ,#'rewrite-template)))

[meain]

I spend a bit of time playing around with it, but I'm not sure how this would work. I was mostly trying out the refactor workflow. I've added the following to the gptel config:

(defun rewrite-template ()
  "Rewrite or refactor selected region"
  (let ((lang (downcase (gptel--strip-mode-suffix major-mode))))
    (list ;; system
          (format (concat "You are a %s programmer.  "
                          "Follow my instructions and refactor %s code I provide.  "
                          "Generate ONLY %s code as output, without "
                          "any explanation or markdown code fences.")
                  lang lang lang)
          ;; user
          (buffer-substring-no-properties (region-beginning) (region-end))
          ;; llm
          "What is the required change?")))


(add-to-list 'gptel-directives '(rewrite . #'rewrite-template))

I selected a section of code, called gptel-menu on it. Set the system directive the rewrite one. Here are the issues that I ran into:

1. If I go to the rewrite menu, I loose the system prompt which was set
2. How should I go about adding the instruction for the refactor?

PS: The inspected list object seems to have the selected section as the final message as well, which I'm assuming was an oversight.

(:model "gpt-4o-mini" :messages
        [(:role "system" :content
                "You are a lisp-interaction programmer.  Follow my instructions and refactor lisp-interaction code I provide.  Generate ONLY lisp-interaction code as output, without any explanation or markdown code fences.")
         (:role "user" :content "(add-to-list 'gptel-directives '(rewrite . #'rewrite-template))
")
         (:role "assistant" :content "What is the required change?")
         (:role "user" :content
                "(add-to-list 'gptel-directives '(rewrite . #'rewrite-template))")]
        :stream t :temperature 1.0)

[meain]

If I set a directive and try to get the lisp object, I get the following error:

Symbol’s value as variable is void: system-extra

I have not spent more time debugging it, just through I would drop this message here for now.

[karthink]

Thanks, should be fixed.

[karthink]

How should I go about adding the instruction for the refactor?

Yes, using a rewrite template this way won't work because there's nowhere to add the instruction. I've updated gptel's rewrite feature to use this template for now, and added the dry-run option so you can see what will be sent. (Run (setq gptel-expert-commands t) to see the dry-run options). Unfortunately the rewrite template is kind of hard-coded -- you can change the gptel-rewrite-directive variable but its new value will have to be another elisp function to be useful, it can't encode the conversation required for rewrites as just a static list of strings.

PS: The inspected list object seems to have the selected section as the final message as well, which I'm assuming was an oversight.

It's not, this is expected since gptel includes the selected region in the prompt by default. This expectation is fine when using the previous way that rewriting worked, where the rewrite instructions were part of the system message and the code from the selected region was part of the prompt. Now both the code and instructions are part of the prompt, as specified by the rewrite template.

---

Overall I'm not yet seeing the benefit of using templates -- it's easy enough to write a wrapper function to do the templating and call gptel inside it. Building templates into gptel inverts this nesting, but doesn't make it any easier to define new useful templates. It looks like templates will have to be dynamic to be useful, and thus require the user to write elisp functions anyway. Going back to my original concerns,

However I am interested in providing the flexibility to create your own templates like the ones in yap, preferably with some UI. This is part of a larger plan to allow gptel to specify an LLM pipeline interactively

Other LLM clients (or web services) offer a plethora of specialized commands like -summarize out of the box, but I find this constraining and prescriptive, and I want to find a way to provide composable building blocks. Hope that makes sense.

The templates as implemented right now should work pretty much how they do in Yap, except that "static" ones with unchanging text can be defined more simply.

[meain]

Unfortunately the rewrite template is kind of hard-coded -- you can change the gptel-rewrite-directive variable but its new value will have to be another elisp function to be useful, it can't encode the conversation required for rewrites as just a static list of strings.

Hmm, I see what you mean. What would make this closer to what yap does would be to provide a system message choice along with directive in refactor as well, but at that point you need to be able to dynamically decide where to plug in the additional user input. Yap completely sidesteps this problem by asking users to define the templates as elisp functions. This way they can fetch in any random bits of information and form a templated chat. All yap provides are utils functions to simplify creating generic templates as well as ways to run the template and perform 3 actions(write, rewrite, prompt).

Which I guess nicely folds into your comments here:

Overall I'm not yet seeing the benefit of using templates -- it's easy enough to write a wrapper function to do the templating and call gptel inside it.

...

It looks like templates will have to be dynamic to be useful, and thus require the user to write elisp functions anyway.

[karthink]

@meain The templating feature is more or less complete, you can test it out now. I plan to merge it in a day or two.

Of the three possibilities for directives: a string (system message), a list of strings (system message + templated conversation) or a function (dynamic system message or conversation), really only the string and function options are useful.

[meain]

Finally got some time to play around with it for longer. I do like the visual interface for picking templates and defining templates.

What I think the major drawback is the fact that I cannot put the user selection and input(directive) into any location within the template. I believe it currently has the restriction of putting user selection only towards the end and the directive into the system message. This might limit the kind of templates that could be built. Also, I think having a separate set of directives potentially for rewrite would help. I understand that one could do it with gptel-rewrite-directives-hook, but I don't know how simple an interface that would be.

I guess all of this could be avoided by writing custom functions and hooking that into gptel, which I believe is basically the core idea in yap.

[karthink]

Templates

What I think the major drawback is the fact that I cannot put the user selection and input(directive) into any location within the template. I believe it currently has the restriction of putting user selection only towards the end and the directive into the system message. This might limit the kind of templates that could be built.

I think I understand the problem here. It's not the template implementation that's lacking, because it works more or less the same as Yap's templates. If you define a template using a function (like Yap does), you can put anything in it, anywhere. For example, here's how you could implement yap-template-split-buffer-context:

(defun gptel-split-buffer-directive (&optional system-prompt)
  (list (or system-prompt (alist-get 'default gptel-directives))
        (format "I'll provide a document with a highlighted section.
The code is in %s.
The answer should be specific to the highlighted section,
but use the rest of the text as context to understand the patterns and intent."
                 (gptel--strip-mode-suffix major-mode))
        "OK.  What is the highlighted text?"
        (buffer-substring-no-properties (region-beginning) (region-end))
        "What is before the highlighted section?"
        (buffer-substring-no-properties (point-min) (region-beginning))
        "What is after the highlighted region?"
        (buffer-substring-no-properties (region-end) (point-max))
        "What can I help you with?"))

(For simplicity I've shown only the case with an active selection.)

The limitation is actually that gptel-send works in an opinionated way: it always and automatically sends the region or the buffer up to point. This can be tweaked using the transient menu but in limited ways.

This makes gptel-send very convenient for back and forth conversation, but not for custom tasks like the ones provided by templates. This is why gptel provides (and is forced to provide) a separate gptel-rewrite command that works differently.

Using an arbitrary template in gptel thus requires dropping into elisp. The above template can be used as:

(gptel-request (read-string "Rewrite instructions: ")
  :system 'gptel-split-buffer-directive
  :in-place t)

which should work exactly like yap-rewrite used with this template.

A one-to-one comparison between yap and gptel works something like this, where relevant-template is any template:

yap-prompt:

(gptel-request (read-string "Prompt: ")
  :system 'relevant-template
  :callback
  (lambda (resp info)
    (if resp
      (with-current-buffer (get-buffer-create "*gptel-response*")
        (erase-buffer)
        (insert resp)
        (display-buffer (current-buffer))))))

yap-write:

(gptel-request (read-string "Prompt: ") :system 'relevant-template)

yap-rewrite:

(gptel-request (read-string "Rewrite instructions: ")
  :system 'relevant-template
  :in-place t)

Alternatively, if the template function itself is responsible for prompting the user (like in Yap), then these calls change to

(gptel-request "" :system 'relevant-template ...)

Making these calls interactive, and allowing the template to be chosen via completing-read will replicate the yap-* commands exactly (modulo some post-processing for generating diffs, accepting the output etc).

---

So I'm not sure what to do. gptel-send's simplicity is also its strength -- it requires a simple mental model, and there's no interaction (completing-reads, menus or prompting) required to use it.

I don't know what the best way would be to allow arbitrary templates to be applied interactively. One way is to define general gptel-* commands, the equivalents of yap-write and yap-rewrite. Let me know if you have any thoughts, I'm holding off on merging the branch into master for now.

Rewrite

Also, I think having a separate set of directives potentially for rewrite would help. I understand that one could do it with gptel-rewrite-directives-hook, but I don't know how simple an interface that would be.

Yeah, I could add a gptel-rewrite-directives option. The advantage of using the hook over this is that the right directive can be chosen automatically based on context. We could have both but that's going to be more confusing to users.

[karthink]

@meain Okay, after thinking about it for a while I've added templates (the full version) to the transient menu. These can be selected

• interactively,
• without requiring custom elisp commands like the above equivalents involving gptel-request,
• and without breaking the current predictable behavior of gptel-send.

Try it out and let me know what you think:

The templates themselves are not included. You can evaluate the test templates here to add them.

---

This brings yap's template functionality in full to gptel, but I think there are some issues with this implementation. I'll wait for your feedback before going into more depth on this.

[meain]

I like the new templating interaction(I guess I'm biased here). I would have personally liked templating for rewrite as well but I can see myself using the templates functionality in gptel instead of yap-prompt for some things now.

One thing I've been primarily been missing from yap is the ability to continue a conversation ideally with added context. For example, I start off a conversation with a template, then continue that in a gptel buffer additionally adding any extra context needed. It would be nice if we had something similar to the current "response to gptel session", but instead of just the last message, it will dump the entire templated chat

Just some notes I had while going through the changes:

• Using nil to skip some interactions in templates looks a bit odd (Not that bad, also this just a personal preference)
  • This is why I went with some way to specify what type of query we are sending in yap, but assuming alternate interactions seems like a simpler approach
    • Another reason was to optionally add in any additional metadata to them messages if necessary. I don't have anything in mind as of now, but I wanted to keep it extendable.
  • Heads up that we cannot use multiple user/bot prompts backup to back in Gemini.
• Is there some way to disable the system message and directive fields if we choose a template? Might make things less confusing to users. Not a huge issue, but just thought I would point it out.
• Just a note about the templates
  • The content under I will provide the following:... is better put as the user message. I've found it to work better this way. Nothing inherently wrong with putting it in system message though.

Unrelated request: We be nice to inspect the query and then send it. Currently inspect dismisses the transient interface.

The advantage of using the hook over this is that the right directive can be chosen automatically based on context.

We might not always be able to pick the right template based on context. For example think if we have two different rewrite options, one to add comments, one to fix bugs and we call rewrite from a prog-mode buffer, it is not something that can be picked automatically. We could use the context to narrow down to particular one though. One possibility I was thinking about was to let people define templates limited to specific major-modes.

[karthink]

I like the new templating interaction(I guess I'm biased here). I would have personally liked templating for rewrite as well but I can see myself using the templates functionality in gptel instead of yap-prompt for some things now.

[...]

The advantage of using the hook over this is that the right directive can be chosen automatically based on context.

We might not always be able to pick the right template based on context. For example think if we have two different rewrite options, one to add comments, one to fix bugs and we call rewrite from a prog-mode buffer, it is not something that can be picked automatically. We could use the context to narrow down to particular one though. One possibility I was thinking about was to let people define templates limited to specific major-modes.

The rewrite system message has been elevated to a first-class directive -- you can pick any from gptel-directives and use it to rewrite, solving the second problem (custom rewrite system messages). There is no separate gptel-rewrite-directives alist, this is intentional. All directives used by gptel -- whether they're used for chat, one-off tasks, templates or refactoring -- live in gptel-directives.

For the first problem (use any template with rewrites), we're almost there but there's one little problem left to sort out:

The rewrite template is currently hardcoded to this:

(system-message  ; <-- Customizable, pick anything from gptel-directives or use the rewrite-hook
 <code here>                    ; <-- hardcoded
 what is the required change?   ; <-- hardcoded
 <rewrite instruction here>)    ; <-- Read from the transient menu
 

It's easy to remove the hardcoding here, but then it becomes more difficult for the user to define a custom template. Instead of specifying just the system message, which they can do simply as a static string, they'll be forced to specify a function that constructs the full list. I'm not sure I want to do that.

One thing I've been primarily been missing from yap is the ability to continue a conversation ideally with added context. For example, I start off a conversation with a template, then continue that in a gptel buffer additionally adding any extra context needed. It would be nice if we had something similar to the current "response to gptel session", but instead of just the last message, it will dump the entire templated chat

It's easy to modify the "response to gptel session" option to dump the filled-in chat template to the gptel session. The only problem is the UI: It will override the current behavior, which is valuable too. Do you think if the directive is a templated conversation (as opposed to a system message only), it always makes sense to dump the whole chat?

Just some notes I had while going through the changes:

* Using nil to skip some interactions in templates looks a bit odd (Not that bad, also this just a personal preference)
  
  * This is why I went with some way to specify what type of query we are sending in yap, but assuming alternate interactions seems like a simpler approach

Yes, I prefer this approach as it has less syntax. You can always add a comment by the side:

'(system                                ;system message
  nil                                   ;user
  "What would you like..."              ;assistant
  "Give me X")                          ;user

    * Another reason was to optionally add in any additional metadata to them messages if necessary. I don't have anything in mind as of now, but I wanted to keep it extendable.

This can be done without much trouble. Extending gptel to handle this is easy:

'(system
  nil
  ("What would you like" :prop1 val1 :prop2 val2)
  ("Give me X" :prop3 val3))

  * Heads up that we cannot use multiple user/bot prompts backup to back in [Gemini](https://github.com/ahyatt/llm/issues/86).

I am aware. The Anthropic API is also quite finicky, but about different things about the prompt.

* Just a note about the templates
  
  * The content under `I will provide the following:...` is better put as the user message. I've found it to work better this way. Nothing inherently wrong with putting it in system message though.

Cool. How do you test this kind of thing? I'm exhausted just thinking about testing all these variations.

Unrelated request: We be nice to inspect the query and then send it. Currently inspect dismisses the transient interface.

This is surprisingly difficult to do right now. I need to rewrite the whole networking code path (required for other features), and I hope to make it easier to pause and resume requests like this.

---

* Is there some way to disable the system message and directive fields if we choose a template? Might make things less confusing to users. Not a huge issue, but just thought I would point it out.

Yes, this is the main drawback I mentioned in my previous message, and the reason I am not merging the templates option into master yet. I'm merging everything else, just not the templates UI from the latest commit.

Fundamentally, there is no difference between a "template" and a "directive" as implemented in gptel. So this distinction is artificial and potentially confusing. If someone selects a directive and then a template, they'd have no idea what is going to be sent. From the perspective of a new-ish user (who is probably aware of what a "system prompt" is), that's too many non-orthogonal concepts required to use gptel.

The difference between the two is only in whether the buffer text is included in the prompt. So I think better nomenclature and UI design is needed -- ideally I'd avoid the term "template" altogether and just make do with directives.

If you have any ideas please let me know.

[meain]

The rewrite system message has been elevated to a first-class directive -- you can pick any from gptel-directives and use it to rewrite, solving the second problem (custom rewrite system messages). There is no separate gptel-rewrite-directives alist, this is intentional. All directives used by gptel -- whether they're used for chat, one-off tasks, templates or refactoring -- live in gptel-directives.

Sounds good to me.

It's easy to remove the hardcoding here, but then it becomes more difficult for the user to define a custom template. Instead of specifying just the system message, which they can do simply as a static string, they'll be forced to specify a function that constructs the full list. I'm not sure I want to do that.

OK, I think the problem is the difference in how we see rewrites. You should just see rewrite as "replace the current text with the response from LLM". Don't necessarily have to think as rewriting code or prose. I think an easy and intuitive(in my subjective opinion) to add an option next to "respond in place" for rewrite. It could be something like "replace selected text" instead of proving a separate rewrite interface. With that in place, the current hardcoded template can be made to be just one of the templates in gptel-templates but somehow highlight that this is useful for rewrites. Yap does this by specifying a "default" template for rewrites.

Do you think if the directive is a templated conversation (as opposed to a system message only), it always makes sense to dump the whole chat?

Not necessarily. Both are useful. It is possible that you might want just the output, but is also possible that you want the full chat. If we really had to pick a way to automatically decide between this, I would say, if it is a new gptel buffer, add full context, otherwise just the response. Otherwise it might be worth it to provide two separate options, but I'm worried if we are just adding too many options now 😅.

Just for some statistics, in my personal use 80-85% of the time, I'm just using the default rewrite template which is similar to what you have for gptel. I use custom template mostly for adding additional info like diagnostics from flymake. It also comes in handy when I need to improve the responses by templating specifically (eg: role prompting).

How do you test this kind of thing?

I don't have any scientific methods for testing any of this. It is more tribal knowledge from randomly experimenting, and lot of papers/blogs/posts.

Fundamentally, there is no difference between a "template" and a "directive" as implemented in gptel. So this distinction is artificial and potentially confusing. If someone selects a directive and then a template, they'd have no idea what is going to be sent. From the perspective of a new-ish user (who is probably aware of what a "system prompt" is), that's too many non-orthogonal concepts required to use gptel.

I like the idea of being able to easily modify system prompt and "directive". That is lost with templates. I don't really have an answers on how to have the same level of ease. One thought I had was to merge system prompt and template that way the current "system prompt" thing will be a template, but with only the system prompt and they can add it in. As for the UI, this would be a separate option in the system prompt picker transient, something like:

s System Prompt

Templates
a Template 1
b Template 2

[karthink]

All commits except the last one adding the "templates" option have been merged into master.

It's easy to remove the hardcoding here, but then it becomes more difficult for the user to define a custom template. Instead of specifying just the system message, which they can do simply as a static string, they'll be forced to specify a function that constructs the full list. I'm not sure I want to do that.

OK, I think the problem is the difference in how we see rewrites. You should just see rewrite as "replace the current text with the response from LLM". Don't necessarily have to think as rewriting code or prose. I think an easy and intuitive(in my subjective opinion) to add an option next to "respond in place" for rewrite. It could be something like "replace selected text" instead of proving a separate rewrite interface.

I don't follow. Have you tried using the Respond in place option? How is it different from the new "rewrite" option you propose?

(I'll refer to the i menu option as the "respond in place" option to avoid confusion with gptel's dedicated "rewrite" interface.)

The point of the dedicated rewrite interface is that it gives you a different UI after the response is received: you can ediff, diff etc. In the beginning I was directing people to the "respond in place" option, but there were many requests for something like the current, dedicated rewrite interface.

Actually, the "respond in place" option also supports ediff. Try the following!

• Select a region of text to rewrite, and an appropriate directive
• Select the (i) Respond in place option and send the request. The text is rewritten in place.
• Now move your cursor into the rewritten region and bring up gptel-menu. You'll see an option to ediff against the original:

• You can also flip through all previous rewritten versions with N and P.

I like this older rewrite workflow, and think it's more elegant and general. But it turned out to be completely undiscoverable, and users kept asking for a dedicated rewrite menu/interface. They also wanted a way to see the proposed changes before they're applied, not after with an option to revert them. Hence the new, loud design with colorful overlays with user-driven changes.

BTW, I remember you mentioned (possibly in a meetup) that gptel's rewrite workflow is too slow for your tastes and has too many steps. Just checking if you're aware of these two things:

1. Once the rewrite overlay is in place, you don't need to bring up the menu again. You can accept/reject/diff/ediff/merge it with one command. You can mouse over the overlay to see this (or turn on Eldoc).
2. There is a gptel-rewrite-default-action option where you can set one of the above actions to occur automatically. The default is to wait for the user to invoke something explicitly.

With that in place, the current hardcoded template can be made to be just one of the templates in gptel-templates but somehow highlight that this is useful for rewrites. Yap does this by specifying a "default" template for rewrites.

This is already the case with Respond in place, as explained above.

Do you think if the directive is a templated conversation (as opposed to a system message only), it always makes sense to dump the whole chat?

Not necessarily. Both are useful. It is possible that you might want just the output, but is also possible that you want the full chat. If we really had to pick a way to automatically decide between this, I would say, if it is a new gptel buffer, add full context, otherwise just the response. Otherwise it might be worth it to provide two separate options, but I'm worried if we are just adding too many options now 😅.

I don't want to populate the menu with even more options, at least for now. Dumping the full prompt if it's a new gptel session sounds like a good compromise to me, I'll do that.

Just for some statistics, in my personal use 80-85% of the time, I'm just using the default rewrite template which is similar to what you have for gptel. I use custom template mostly for adding additional info like diagnostics from flymake. It also comes in handy when I need to improve the responses by templating specifically (eg: role prompting).

You can do this even with the current implementation of the hardcoded rewrite interface:

(defun gptel--rewrite-with-diagnostics ()
  (list "You are a ... programmer.  Follow instructions..." ;system
        (concat "Here are the relevant linter errors :"
                (collect-flymake-diagnostics-here))         ;user
        nil))                                               ;assistant

(add-to-list 'gptel-directives
             '(rewrite-with-diag . gptel--rewrite-with-diagnostics))

This is prepended to the hardcoded prompt to give you

(list "You are a ... programmer.  Follow instructions..."  ;system
       "Here are the relevant linter errors:
        - error 1
        - error 2
        ..."                                               ;user
       nil                                                 ;assistant
       ;; The following is the current hardcoded template
       (list (buffer-substring-no-properties
               (region-beginning) (region-end))            ;user
        "What is the required change?"                     ;assistant
       (or rewrite-message gptel--rewrite-message)))       ;user

Of course, this is a brittle sort of composition and we need a better solution.

I like the idea of being able to easily modify system prompt and "directive". That is lost with templates. I don't really have an answers on how to have the same level of ease. One thought I had was to merge system prompt and template that way the current "system prompt" thing will be a template, but with only the system prompt and they can add it in. As for the UI, this would be a separate option in the system prompt picker transient, something like:

I don't understand this idea. Here's how I'm using these terms:

• system-prompt == system-message :: A string, fed to the LLM as its system prompt.
• directive == template :: A list of strings. The first string is the system prompt and the rest are a back and forth conversation. Any list element (including the system prompt) can be nil. This term is overloaded to also mean a function that returns such a list of strings.

the current "system prompt" thing will be a template, but with only the system prompt and they can add it in.

This is not clear. They can add what in?

[meain]

Have you tried using the Respond in place option? How is it different from the new "rewrite" option you propose?

Ohh, this seems useful. I had misunderstood what what it did. I was also totally unaware of "Tweak Response". Seems like a great alternatively to having a separate rewrite interface to me. To be frank I did not spend a lot of time trying to play around with gptel initially before building out yap as I had found quite few ideas that I wanted to experiment with which would have been hard with gptel.

I like this older rewrite workflow, and think it's more elegant and general. But it turned out to be completely undiscoverable, ...

Agreed on both of those points, about it being elagant and about it being undiscoverable. About the discoverability issue, one small change might be to change "respond in place" to "rewrite selection" if there is an active selection.

BTW, I remember you mentioned (possibly in a meetup) that gptel's rewrite workflow is too slow for your tastes and has too many steps. Just checking if you're aware of these two things:

They also wanted a way to see the proposed changes before they're applied ....

The ideal workflow for rewrites for me(and what is currently in yap) is to see the output of the llm while keeping the original intact, if necessary, see a diff and then apply if it looks good. Viewing the diff after the fact is an OK compromise. If we don't already, providing something like gptel-rewrite-default-action would be nice when they use "respond in place" as well.

I also kinda like what chatgpt-shell does where the new output is added along with the current one in the same buffer(unlike yap where we open it in different buffer). This might however get tricky if you are doing big(over 50 lines) rewrites which I tend to do ~25% of the time.

... loud design with colorful overlays with user-driven changes.

This is the part that I'm not a big fan of. In most cases, when I have performed a rewrite, I don't want to go back and then invoke gptel-menu again to apply it. IIUC, I could use gptel-rewrite-default-action to automatically apply or view diff, but I would just like to see the output separately. The current gptel workflow might make sense for slow models where waiting for output is annoying. Being able to stream response and view the output as it comes from the llm is super useful which is not available in the current rewrite workflow with gptel IIUC. Streaming also kinda "mitigates" the problem with slow models.

You can do this even with the current implementation of the hardcoded rewrite interface:

Yup, I understand. I just wanted to point out what my current use-case was like.

I don't understand this idea. Here's how I'm using these terms:

I don't think I communicated what I was thinking coherently. What I meant was that it would nice if we could define "variables" within the template which could be visually filled in via the template the same way we do with system and directive entries. For example, in the YouTube example that you had, the transient would add a "button" for the URL and when you fill it in, you can see the url in the UI before you hit "RET". But the more I think about it, the more if feels like a unnecessary complication.

[meain]

Since we are talking about templates, I wanted to bring your attention to the possibility of loading templates from outside of the elisp. This can always be just an additional function at a later point, but just throwing the idea out there. A lot of tools/services are popping up now which lets uses create template collections.

Tracking ticket in yap: meain/yap#20

[karthink]

Cool, this is pretty easy to add at any point. If the .cursorrules file becomes a de facto standard I can support it in gptel.

[karthink]

The ideal workflow for rewrites for me(and what is currently in yap) is to see the output of the llm while keeping the original intact, if necessary, see a diff and then apply if it looks good. Viewing the diff after the fact is an OK compromise.

I couldn't get yap-rewrite to run because of a dependency issue with llm's dependencies (something to do with plz-event-source that I haven't worked out yet.) So I'm just imagining how it works based on reading the source, since I couldn't find any gifs/videos either.

Here's what I think you're suggesting

1. Wait for a streaming response to a rewrite request to start.
2. When it starts, open up a new window and stream the response to it.
3. Set some keybindings in this window to accept the response or diff against it.
4. After it's done streaming in, wait for the user to invoke one of the available actions.

The difference between this and how gptel-rewrite currently works is that

1. Instead of streaming the response, gptel waits for the whole thing to be received.
2. Instead of showing it in a new window, gptel hides it in an overlay placed over the original text, so you can't see the response without invoking an action first (ediff, diff, merge or accept).

Does that cover it, or did I miss something?

If we don't already, providing something like gptel-rewrite-default-action would be nice when they use "respond in place" as well.

I'm not sure how this would work? The original text is deleted when the response starts streaming in. (It remains available via the "Tweak response" options)

This is the part that I'm not a big fan of. In most cases, when I have performed a rewrite, I don't want to go back and then invoke gptel-menu again to apply it.

You don't have to go back to the menu. You can invoke all available actions directly:

I also kinda like what chatgpt-shell does where the new output is added along with the current one in the same buffer(unlike yap where we open it in different buffer). This might however get tricky if you are doing big(over 50 lines) rewrites which I tend to do ~25% of the time.

This looks like the same as (setq gptel-rewrite-default-action 'merge) to me. Since gptel-rewrite-default-action
has three more viable actions (automatically diff, ediff or accept) it's actually more flexible than this.

What I meant was that it would nice if we could define "variables" within the template which could be visually filled in via the template the same way we do with system and directive entries.

Hmm, I need to think about this some more.

[meain]

I couldn't get yap-rewrite to run because of a dependency issue with llm's dependencies (something to do with plz-event-source that I haven't worked out yet.)

Huh, let me look into it. You are correct in the assumption. In any case, here is a demo. Also, would it be possible to try installing llm package separately?

yap-rewrite-example.mov.mp4

You don't have to go back to the menu. You can invoke all available actions directly:

True, but streaming in the response let's me start validating the response as it streams in and for most paid services, the tokens per second are more than my reading speed that this makes for a good workflow. This way, I don't have to sit there doing nothing until the model response is fully available.

Just a small note on the UI. "Refactor" seems too specific. "REWRITE READY" might be a more better term to use in the place of "REFACTOR READY". Also, probably don't insert "Refactor:" into the input box. It could be a generic set of instructions. Again, all are personal preferences. Just letting you know how I would have gone about it. In fact, I think rewrite too is a bit too specific that I'm thinking about switching to replace in yap as I would also want to support images(if I ever get to doing it is a separate queestion). I might want to provide an image as input, as it to "replace" the image with a modified version of it.

This looks like the same as (setq gptel-rewrite-default-action 'merge) to me. Since gptel-rewrite-default-action has three more viable actions (automatically diff, ediff or accept) it's actually more flexible than this.

Again, my bad. I misunderstood what merge did. This looks good, but the problem I mention about having to review longer replaces still exists.

[karthink]

Just a small note on the UI. "Refactor" seems too specific. "REWRITE READY" might be a more better term to use in the place of "REFACTOR READY". Also, probably don't insert "Refactor:" into the input box. It could be a generic set of instructions.

Removed "refactor" from all menus and indicators.

[karthink]

Ohh, this seems useful. I had misunderstood what what it did. I was also totally unaware of "Tweak Response".
[...]
Again, my bad. I misunderstood what merge did

My main takeaway from this thread is actually that I'm doing a very poor job of communicating gptel's features. For the longest time I was having trouble explaining to users that gptel works anywhere -- they assumed you had to open a special "chat" buffer. Perhaps they still do.

Trying to make features like rewriting intuitive without hitting them in the face with it is going to be difficult!

The current gptel workflow might make sense for slow models where waiting for output is annoying. Being able to stream response and view the output as it comes from the llm is super useful which is not available in the current rewrite workflow with gptel IIUC. Streaming also kinda "mitigates" the problem with slow models.

I have one point of disagreement and one of agreement.

• Once I fire off the rewrite I go do something else, and come back to it at my convenience. In this context, I'd find another window opening up quite distracting. In general my goal for gptel is to have a small footprint -- so no manipulating the Emacs frame if we can help it.

• However, I agree that being able to view the response without any effort is important. One annoying aspect of gptel's current rewrite method is that you can't just read the response once it's ready. The overlay is placed over the existing text, and you have to invoke a command to see the response. As I don't want gptel to pop up another window, I'm trying something like this:

gptel-rewrite-in-place.mp4

Now you can see the response as it comes in, and the overlay with the same diff/ediff/merge/apply actions is available after it's done. It also avoids the problem with long text chunks that you get from auto-using the merge option (the chatgpt-shell feature).

What do you think of this approach?

The big problem with this approach is that the incoming text needs to actually be inserted into the buffer -- which the user may not want. I can do this trick with overlays alone and not modify the buffer, but the rewritten region will then be intangible, which causes other issues.

[meain]

My main takeaway from this thread is actually that I'm doing a very poor job of communicating gptel's features.

I also probably didn't do my fair share of reading of gptel docs.

I was having trouble explaining to users that gptel works anywhere -- they assumed you had to open a special "chat" buffer. Perhaps they still do.

One suggestion I would like to make is to add a gif at the top for refactor/rewrite. The top gifs are currently just about having a separate chat session. People(including me) don't generally even read the entire README.md.

Once I fire off the rewrite I go do something else, and come back to it at my convenience.

I feel like it is a difference in workflow. Since I'm mostly working with code, as soon as I see about 10% of the response, I can mostly decide if the response is worth waiting for or if I should just ignore this one and think of a better prompt. And for this, I need to see the response as it streams in.

What do you think of this approach?

This seems to be better than the current approach. This way at least we can see what the llm is coming up with as it is coming up with it. I don't know if I'll be able to halfway through give up on the current output and decide to try another prompt.
FYI. I don't think this code is pushed to the branch yet. Does not look like I'm getting the new behavior.

The big problem with this approach is that the incoming text needs to actually be inserted into the buffer -- which the user may not want.

I agree. In my workflow, I need to be able to know if the thing that the llm produces is useful before it is complete, but I ideally don't want it to change anything in my buffer. That said, most other tools that do refactor, either provides an inline diff or rewrite the content and so this is definitely an improvement. Since the users can open a diff once the rewrite it complete, I think the workflow is good enough. Will have to do a good job at communicating about good workflows.

I can do this trick with overlays alone and not modify the buffer, but the rewritten region will then be intangible, which causes other issues.

I too feel that that will be more annoying than useful.

[karthink]

One suggestion I would like to make is to add a gif at the top for refactor/rewrite. The top gifs are currently just about having a separate chat session. People(including me) don't generally even read the entire README.md.

Yeah, I'll need to do this. One thing I did in gptel-rewrite just now is hit the user over the head with the available options, probably to the point of annoying them. From the commit message:

gptel-rewrite: We are now bombarding the user with hints from
every direction: Eldoc, minibuffer messages, the transient menu,
mouseover text (help-echo), and now a dispatch command invoked
via RET or clicking the overlay.  Hopefully that's enough!

What do you think of this approach?

This seems to be better than the current approach. This way at least we can see what the llm is coming up with as it is coming up with it. I don't know if I'll be able to halfway through give up on the current output and decide to try another prompt. FYI.

Making it easy (one keybind) to abort rewrites is planned.

I don't think this code is pushed to the branch yet. Does not look like I'm getting the new behavior.

It was a prototype demo. I pushed it now, although it works somewhat differently from the demo (see below).

The big problem with this approach is that the incoming text needs to actually be inserted into the buffer -- which the user may not want.

I agree. In my workflow, I need to be able to know if the thing that the llm produces is useful before it is complete, but I ideally don't want it to change anything in my buffer.

The more I thought about it the worse the idea seemed -- so I've switched to the full-overlay method now. Between auto-formatters, LSP, linters and other minor-modes,I think modifying the buffer can have all kinds of undesirable side effects, depending on what the user is running.

I can do this trick with overlays alone and not modify the buffer, but the rewritten region will then be intangible, which causes other issues.

I too feel that that will be more annoying than useful.

This is the approach I'm using now (please test). I've made it easier to accept/reject the changes, so I'm hoping that offsets the overlay tangibility problem. While it's not entirely intangible, it takes a little more care to make sure the point is in the overlay now. Hopefully that'll work.

gptel-rewrite-code-demo-1.mp4

[meain]

gptel-rewrite: We are now bombarding the user with hints from
every direction: Eldoc, minibuffer messages, the transient menu,
mouseover text (help-echo), and now a dispatch command invoked
via RET or clicking the overlay. Hopefully that's enough!

Hahaha. Hopefully that is enough.

Between auto-formatters, LSP, linters and other minor-modes,I think modifying the buffer can have all kinds of undesirable side effects, depending on what the user is running.

Huh, I did not think about it. Makes sense.

This is the approach I'm using now (please test).

Seems to work well enough.

[karthink]

Aborting rewrites in progress now works correctly with gptel-abort:

screencast_20241202T234523.mp4

I think the rewrite interface is in much better shape now. There's only one problem: the overlay display becomes wonky when it's larger than the screen height. It still works but will probably be confusing to users.

[meain]

Aborting rewrites in progress now works correctly with gptel-abort

Sweet!

There's only one problem: the overlay display becomes wonky when it's larger than the screen height. It still works but will probably be confusing to users.

Any time I've worked with overlays, it works great in the beginning, but then you end up spending a lot of time fixing all the edge cases.

While we are on the topic of rewrites. I've seen a few tools where they get a code response along with an explanation from the llm model, but use just the first code block for rewrite or in general do some post processing on the output received from the llm to get to what the rewrite text is. I'm not sure how we would want to model that interaction. I've been thinking more about this as I've had hard time making some llms follow the instruction "only give me the code, no explanations". In the particular case of code with explanations scenario, I've seen the explanation go into a separate buffer but with the code going into where we would replace it. I think it should be easy to extend the package to do it at a later point if we want to. Not something that we should block on for "releasing" the new rewrite interface, but thought I would mention it.

[meain]

Just for reference, this is what Zed does. It is really good. I don't know how tricky it will be to get a similar flow working with Emcas overlays. This UI is super useful when there is only a few additions and we are not rewriting the entire thing.

Screen.Recording.2024-12-03.at.8.39.44.PM.mov

[karthink]

Implementing this with overlays in gptel isn't much of an issue, but this UI is very confusing! I had to watch it several times to understand what was happening. I think the problem is that they're doing two kinds of diff and switching the display out from under you, creating a lot of visual noise -- there's a coarse diff as the response is coming in, and a refined diff once the response is done. If you're following the stream, there's a jarring moment when the diff switches from coarse to fine that your eyes can't track.

Until you accept the response, are the changes virtual text/overlays, or are they in the buffer?

I understand why they're doing a coarse diff while the response is streaming though. If you had to diff everything received so far line-wise or word-wise against a suitable subset of the original text every time you received a response chunk, you'd be doing too many diff operations. Zed can probably manage that without trouble, actually, but there's no way it would run smoothly in Emacs.

[meain]

they're doing two kinds of diff and switching the display out from under you

Ahh, true. I think this is one of the worst cases of diff issue that I've seen though. Most of the time, it ends up looking pretty good since I request for small changes in the code.

Until you accept the response, are the changes virtual text/overlays, or are they in the buffer?

Something like overlays.

Zed can probably manage that without trouble, actually, but there's no way it would run smoothly in Emacs.

Hmm, that's unfortunate.

[karthink]

I think we can improve the current rewrite previews to display more granular changes, but not quite how Zed does it. Not in the upcoming release though, I'll get to it eventually.

[karthink]

I'm planning to tag a new release with the improved rewrite UI and the generalized directives features.

Templates aren't in yet. As I mentioned above, they're the same things as directives but used differently, and I haven't found a good way to communicate this to the user yet. This screen crosses a confusion threshold for me:

There's a system message option, a "directive" option below that, and a "prompt from template" option, and they're all related. So this UI needs some work. Maybe there is some reshuffling/renaming of these options that can make it less confusing.

Templates have one more problem -- they're non-interactive functions that query the user interactively. While this practice won't send the Elisp cops after us, it is considered bad design in Emacs. FWIW I'd like to avoid this too.

Do you have any suggestions for me with regards to these features before I tag the release?

[meain]

... generalized directives features

What exactly do you mean by generalized directives features?

There's a system message option, a "directive" option below that, and a "prompt from template" option, and they're all related. So this UI needs some work. Maybe there is some reshuffling/renaming of these options that can make it less confusing.

Yeah, I agree that this can be a little confusing. IMO, I have not been a big fan of the current implementation of directives being just appended to the system message. I've never find it to be as useful as just providing a separate user message. I think directive should always go in as the first user message. If they really want to get the current behavior(which I don't think is useful), they can just edit the system prompt and add text at the bottom.

Along with that, I think we should just unify "Instructions" and "Prompt from". I'm thinking something like:

Instructions:                Directive:                        Context
  s  Set system message      d  Add instruction (none)         -b Add a buffer to context
  t  Set template            u  From kill-ring                 -f Add a file to context

Directive is not the right word here, but yeah. Maybe "user instruction"?

The "Add directive" and "Prompt from minibuffer instead" merges into "Add instructions". With this we can remove "Prompt from" completely. Along with that, we could move "Respond in place" to under "Response to".

The "Reponse to" can be changed to "Response" and this is the new list:

• in place
• minibuffer
• kill-ring
• gptel session
• any buffer
• ask buffer (I don't know what this is. Can we combine this with any buffer?)

Also, regarding the context. Context is better kept at the top. In this case, keep it before directive which is not what is done right now.

Templates have one more problem -- they're non-interactive functions that query the user interactively. While this practice won't send the Elisp cops after us, it is considered bad design in Emacs. FWIW I'd like to avoid this too.

This is a tricky one. I don't really have any good ideas here. We could maybe ask people to mark their templates where we should fill in a variable via a prompt and parse it and display. I don't know if it is a good idea, but just wanted to point out the option. It is possible that they would want to do more preprocessing on the received info, but only if we are actually sending the request.

[karthink]

Thanks for the suggestions, they've got me thinking.

What exactly do you mean by generalized directives features?

The meaning of "directive" changed, from

system message string(S) to

union(S, S+Conversation(C), f: _ -> union(S, S+C)).

I have not been a big fan of the current implementation of directives being just appended to the system message. I've never find it to be as useful as just providing a separate user message. I think directive should always go in as the first user message. If they really want to get the current behavior(which I don't think is useful), they can just edit the system prompt and add text at the bottom.

This is easy to fix, but I wonder if this works correctly all the time, i.e. in all situations where you can slap on an additional instruction to the system message. I'll change this and we can see how it goes.

---

Along with that, we could move "Respond in place" to under "Response to".

That's where it belongs, I kept it separate mainly to balance the column heights.

Along with that, I think we should just unify "Instructions" and "Prompt from".

I like the symmetry of the "Prompt from" and "Response to" sections, it's like input and output redirection at the shell.

The "Reponse to" can be changed to "Response" and this is the new list:

Sure. You can ignore the "ask buffer" option, it's a personal customization.

Incrementally, so far we have:

• I've made the input/output redirection idea more mnemonic with shell-style redirection symbols: <Prompt from, >Response
• I've moved the in-place option over to the Response column.

---

The "Add directive" and "Prompt from minibuffer instead" merges into "Add instructions".

"Add directive" and "Prompt from minibuffer" do very different things. The former changes the "instructions", the latter changes the "data". In keeping with the shell analogy, gptel-send works like this by default:

cat /dev/stdin | gptel-send --system $system_message

"Add directive": changes the system message, but leaves the rest of the constructed prompt alone.

cat /dev/stdin | gptel-send --system $(echo $system_message $add_directive)

Even if I change add-directive to add to the first user prompt instead of the system message, it becomes

{echo $add_directive; cat /dev/stdin} | gptel-send --system $system_message

"Prompt from minibuffer": replaces the prompt entirely with what is read from the minibuffer:

prompt-from-minibuffer | gptel-send --system $system_message

(No use of the buffer == stdin here)

If you use both you get

prompt-from-minibuffer | gptel-send --system $(echo $system_message $add_directive)

or

{echo $add_directive; prompt-from-minibuffer} | gptel-send --system $system_message

So it does not make sense to merge them.

---

I'll address your other suggestions soon.

[meain]

This is easy to fix, but I wonder if this works correctly all the time, i.e. in all situations where you can slap on an additional instruction to the system message. I'll change this and we can see how it goes.

I spent more time thinking about it and I think the part that felt odd to me is that you would want to edit the system message using directives. While I think it is useful to have, I feel that it could be confusing especially because we have the option to edit the system message IIRC.

I don't think I was clear in my previous explanation. The idea was not to put the directive in the user message, but to remove the option for directive and replace with "prompt from minibuffer" but accept and show it in the transient UI.

I would rather remove the direction option there and put the "template" option instead as it does not belong along with the other options which are for getting the user message for the selected system message.

[karthink]

I don't think I was clear in my previous explanation. The idea was not to put the directive in the user message, but to remove the option for directive and replace with "prompt from minibuffer" but accept and show it in the transient UI.

I'm sorry, this is still not clear. "prompt from minibuffer" is not related to the "additional instruction/directive" option at all, for the reasons explained using the shell analogy in my above post. Replacing the "additional instruction" option with "prompt from minibuffer" will leave the user with no way to add instructions for acting on the buffer text.

I would rather remove the direction option there and put the "template" option instead as it does not belong along with the other options which are for getting the user message for the selected system message.

You're correct that the template overwrites the system message, so it's not on par with the other "Prompt from" options, and doesn't belong there. However the template combines both the message and the prompt, so putting it in the system message sub-menu doesn't make sense either, for the same reason.

I guess the problem is that gptel treats the system message and the prompts as independently composable things, but the idea of a task template bakes in the two so it doesn't fit anywhere in the menu.

[meain]

I'm sorry, this is still not clear. "prompt from minibuffer" is not related to the "additional instruction/directive" option at all, for the reasons explained using the shell analogy in my above post. Replacing the "additional instruction" option with "prompt from minibuffer" will leave the user with no way to add instructions for acting on the buffer text.

Let's assume the user want to create the following request:

(:model "gpt-4o-mini" :messages
        [(:role "system" :content
                "You are a large language model and a careful programmer. Provide code and only code as output without any additional text, prompt or note.
Write simple code")
         (:role "user" :content "Write python code to list mangos")]
        :stream t :temperature 1.0)

This was achieved by doing starting with the programmer system prompt, then adding a directive "Write simple code" and then providing the message "Write python code to list mangos" via the minibuffer. I'm just saying that the user can achieve the same result by just editing the system message directly and adding it instead of adding a "directive". The bigger point here being the fact that any smaller request specific instructions are better provided as a user message and not appended to the system message.

I guess the problem is that gptel treats the system message and the prompts as independently composable things, but the idea of a task template bakes in the two so it doesn't fit anywhere in the menu.

The way I view it, system message is also just a template, but a template with only a single(system) message, and one that accepts the user message via the UI (if necessary) instead of a read function in the template. But yeah, I do agree with what you are saying.

[karthink]

Unrelated request: We be nice to inspect the query and then send it. Currently inspect dismisses the transient interface.

0f4136e

Dry run output can now be freely edited by hand and the request continued. The original request specification (callback etc) is respected.

[meain]

Sweet! 🙏🏼

[karthink]

Heads up that we cannot use multiple user/bot prompts backup to back in Gemini.

I just tried this with Gemini and it worked fine, so not sure what's going on. (Context: I'm trying to simplify gptel's prompt construction process and being able to append back-to-back user (or assistant) messages really helps)

[meain]

While I had not explore it, I was just gonna merge(string-join with "\n\n") back to back user messages if for some reason the llm was not supporting it. Glad I don't have to do that with Gemini.

[karthink]

@meain In case you're interested: #514