Change the Prompt Message

[seiggy]

Change the Prompt Message to use the "system" role for better accuracy.

I've found (at least with my experiments with PF2e) that if you send nothing for the system message and 2 user messages, you'll get less accurate responses where the system tends to invent rules that don't exist. This change should tighten up rules accuracy a little in PF2e, and hopefully shouldn't negatively impact other systems (a couple cursory tests with D&D 5e questions seemed just fine).

Here's the sample results I have for the 2 different approaches:

Using "user" for both messages, with the question: "What are the rules for grappling?"

Response:

Here are the rules for grappling in Pathfinder 2nd Edition:

1. Declare the Grapple: The attacker must declare that they are attempting to grapple before making the attack roll.
2. Make the Attack Roll: The attacker rolls an Athletics check against the target's Fortitude DC.
3. Success: If the attacker's Athletics check result is greater than or equal to the target's Fortitude DC, the target is grabbed and both creatures are now grappling.
4. Failure: If the attacker's Athletics check result is less than the target's Fortitude DC, the grapple attempt fails and the attacker's turn ends.
5. Maintain the Grapple: On subsequent turns, the attacker can attempt to maintain the grapple by making another Athletics check against the target's Fortitude DC. The target can attempt to break free by using an Athletics or Acrobatics check against the attacker's Athletics DC.
6. End the Grapple: The grapple ends if the attacker releases the target, the target escapes, or the attacker is incapacitated.

Note that grappling imposes the grabbed condition on both creatures, which can limit their actions. Also, some creatures may have special abilities

Using "system" for the system prompt and "user" for the user's question:

Response:

To start a grapple, you must first succeed at a Athletics check against your target's Fortitude DC. If you succeed, you and your target become grabbed. While grabbed, you have the immobilized condition and can't use any actions with the attack or manipulate traits that require two hands. You can use the Grapple action to move your target or damage them with your Athletics skill, or you can release the grab as an Interact action. Your target can use the Escape action to try to break free, using either Athletics or Acrobatics against your Athletics DC. Additionally, if you're grabbed by a creature, you can try to break free using the Escape action. If you succeed, you free yourself and the creature loses the grabbed condition. If you critically succeed, you get away and the creature is flat-footed until the end of your turn.

The key important differences here are the addition of the immobilized condition, the explanation of that condition, and occasionally when using "user" ChatGPT will claim you can do things like "move" or "pin" a creature, which are not allowed actions in PF2e. I have yet to see said issue when using the "system" approach.

[vizovitin]

Thanks for your contribution! And thank you specifically for providing query and response examples.

Which GPT model version are you using?

During the initial implementation I also used system role for the initial prompt, but found that at least GPT-3.5 tends to loose focus on it. Anyway, I'll test your suggestion later in more detail.

Since I'm not that familiar with PF2e, it would also help me to have a number of queries handy, if somebody is willing to spend time to come up with them.

[seiggy]

Oh, I was using GPT 3.5 turbo

[seiggy]

Sure thing! I'll build out a bunch of test queries to help. Might need to add a switch to use system role for PF2e and user for 5e if it starts to loose focus for 5e. I didn't test much with GPT4 because it's so much slower and more expensive.