Skip to content

Commit

Permalink
Fix prompt format mismatch with huggingface
Browse files Browse the repository at this point in the history
1. System prompt: remove <s> => token [1] is generated by default.
2. End of System prompt:
Before:         -> After
\n\n               \n\n"""
"""
=> Origin code implies three \n.
3. Fix append_user_prompt & append_bot_prompt to match behavior of
   `tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)`
Correct Format for LLama2:
```
<s>[INST] <<SYS>>
{{ system_prompt }}
<</SYS>>

{{ user_msg_1 }} [/INST] {{ model_answer_1 }} </s><s>[INST] {{ user_msg_2 }} [/INST]
```

Signed-off-by: Bruce Lai <bruce.lai@sifive.com>
  • Loading branch information
bhbruce committed Aug 7, 2024
1 parent 26ce08e commit 71d671c
Showing 1 changed file with 6 additions and 8 deletions.
14 changes: 6 additions & 8 deletions models/turbine_models/custom_models/llm_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,27 +61,25 @@
parser.add_argument(
"--chat_sys_prompt",
type=str,
default="""<s>[INST] <<SYS>>
Be concise. You are a helpful, respectful and honest assistant. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n <</SYS>>\n\n
""",
default="""[INST] <<SYS>>
Be concise. You are a helpful, respectful and honest assistant. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n<</SYS>>\n\n""",
help="System prompt used for interactive chat mode.",
)

B_INST, E_INST = "[INST]", "[/INST]"
B_SYS, E_SYS = "<s>", "</s>"
DEFAULT_CHAT_SYS_PROMPT = """<s>[INST] <<SYS>>
Be concise. You are a helpful, respectful and honest assistant. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n <</SYS>>\n\n
"""
DEFAULT_CHAT_SYS_PROMPT = """[INST] <<SYS>>
Be concise. You are a helpful, respectful and honest assistant. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n<</SYS>>\n\n"""


def append_user_prompt(history, input_prompt):
user_prompt = f"{B_INST} {input_prompt} {E_INST}"
user_prompt = f"{input_prompt} {E_INST}"
history += user_prompt
return history


def append_bot_prompt(history, input_prompt):
user_prompt = f"{B_SYS} {input_prompt}{E_SYS} {E_SYS}"
user_prompt = f"{input_prompt} {E_SYS}{B_SYS}{B_INST}"
history += user_prompt
return history

Expand Down

0 comments on commit 71d671c

Please sign in to comment.