Curious about the off policy part on pi_{phi}

[zpointS]

Hi Jianhao and other authors,
Thank you for your wonderful job!
I'm a little bit confused on how you actually handle the pi_{phi} part in off-policy loss. Sorry I'm not very familiar with RL.
It is said in paper that you adopted pi_{phi} == 1, so does that mean the off policy objective simply become the average value of pi_{theta} * adv over all tokens of all samples (ignoring other caliberating parts)?
I'm wondering if you are actually using log(pi_{theta}) in realization because modern LLMs keep telling me although we are given pi_{phi} == 1, we should use logp instead of raw p for pi_{theta} as a common practice in policy gradient. But PG is different from PPO and GRPO, and that's what makes me confused.
Thanks!

[ElliottYan]

Thanks for your interest in our work!
The pi_phi part does not change the policy gradient derivation. In fact, log p_theta appears in the gradient computation. When taking the derivative of \sum p_theta / p_phi * adv -> \sum stop_grad(p_theta / p_phi) * adv * \delta log p_theta. That's why in modern implementation of ppo and grpo does not actually use log P in their objectives.

[zpointS]

Thank you for replying!
So briefly, what you did is setting p_phi == 1 when its off policy, is that right? Can you please show me where did you make p_phi == 1 in your code? Cuz it seems like it is not explicitly written / hardcoded. Is it in the batch of training data?