Theoretical Analysis and Idea of SPIN are quite weird (may not make senses)??

[RL4LLM]

1. At iteration 0, the p_{\theta_0}=p_{SFT} and the global optimal p_{\theta_1} after iteration 1 of following objecitve will still be p_{SFT} . Thus, the following iterations of p_{\theta} will always be p_{SFT}.

3. In your theoretical analysis, you also prove that p_{\theta*}=p_{data}; however, p_{SFT}=p_{\theta*} by minimizing the cross-entropy. I believe the reason why SPIN outperforms p_{SFT} is not clear. Could you please explain this?