Question regarding <think> tag during online inference

[demon2036]

Thanks for open-sourcing the code and data. Quick question on the inference process:

During online inference, does the model output include the tag? I would like to confirm if the model generates the reasoning/thinking traces (wrapped in tags) by default when serving the request, or if this is stripped out?

Thanks!

[zzx-zev]

Thanks for your interest in our work. And, yes, the output of the model include tag by default.

If you want to close thinking mode, there are two different ways:

1. Soft switch: append the /no_think sequences to the end of your user prompt.
2. Hard switch: set enable_thinking=false when calling apply_chat_template (for transformers-like libs), or add an empty thinking trace ("<think>\n</think>\n\n") directly at the begin of assistant response.

You can find more detailed information from Qwen3: switching-between-thinking-and-non-thinking-mode.

[demon2036]

Thanks for your interest in our work. And, yes, the output of the model include tag by default.

If you want to close thinking mode, there are two different ways:

1. Soft switch: append the /no_think sequences to the end of your user prompt.
2. Hard switch: set enable_thinking=false when calling apply_chat_template (for transformers-like libs), or add an empty thinking trace ("\n\n\n") directly at the begin of assistant response.

You can find more detailed information from Qwen3: switching-between-thinking-and-non-thinking-mode.

Thanks for the clarification!

So, just to confirm: online inference has the thinking mode enabled by default.

Regarding the thinking length, is it limited to 1024 tokens? If so, is the additional latency introduced by this thinking process generally considered acceptable for online/production serving?

[zzx-zev]

Thanks for your interest in our work. And, yes, the output of the model include tag by default.
If you want to close thinking mode, there are two different ways:

1. Soft switch: append the /no_think sequences to the end of your user prompt.
2. Hard switch: set enable_thinking=false when calling apply_chat_template (for transformers-like libs), or add an empty thinking trace ("\n\n\n") directly at the begin of assistant response.

You can find more detailed information from Qwen3: switching-between-thinking-and-non-thinking-mode.

Thanks for the clarification!

So, just to confirm: online inference has the thinking mode enabled by default.

Regarding the thinking length, is it limited to 1024 tokens? If so, is the additional latency introduced by this thinking process generally considered acceptable for online/production serving?

I see what you mean. This work mainly focuses on offline benchmarks. For online A/B tests, we’re still evaluating across multiple KuaiShou online scenarios, and the thinking mode configuration is different depending on the product setup and constraints. Since the experiments are still in progress, we don’t have any solid result to share yet.

But, based on our previous work OneRec-Think, I think a Think-Ahead style approach might be a good method for latency, using a hybrid offline + online generation pipeline, which can support more than 1024 tokens as CoT while keeping latency within a controllable budget.