Qualitative results trained on single reward / Head-to-head comparison with multiple rewards

[jaylee2000]

Thanks for putting out inspiring work!

I was wondering if you are willing to release qualitative results and/or checkpoints of DiffusionNFT trained on a single reward. In my opinion, the most impressive result of DiffusionNFT is its head-to-head comparison with FlowGRPO (Figure 6), made when trained on a single reward. However, all qualitative results (Figures 5, 11-13) & evaluation on multiple rewards (Table 1) are based on the multi-reward optimized model. I am concerned that the huge acceleration in the single reward setting may have accompanied more severe reward hacking than Flow-GRPO.

I am also curious if you are willing to add a head-to-head comparison against Flow-GRPO in multi-reward joint training. I would like to know whether the huge acceleration and perfomance gap are still prevalent.

[ChenDRAG]

Thank you for your interest in our work.

We will add qualitative results for head-to-head comparisons soon. Our early experiments show that when DiffusionNFT and FlowGRPO are both initialized from a CFG-free model (head-to-head), DiffusionNFT is better than FlowGRPO in terms of reward hacking, efficiency, and final performance. Initializing from the CFG provides help in terms of generalization and diversity, especially when using dedicated single-task reward models such as OCR.

Adding multi-reward comparison against Flow-GRPO is difficult because we run multiple sequential experiments to iteratively improve performance on different reward models. It is hard to control training time and scheduling of switching among different reward models.

[jaylee2000]

Amazing! Do you plan on uploading checkpoints for single-reward settings too? It would be very helpful to compare DiffusionNFT to previous methods, which are mostly trained on single rewards as well.

By the way, could you clarify what “initializing from CFG / CFG-free” means?

[ChenDRAG]

CFG-free means the guidance_scale = 1.0 throughout training and evaluation (our default behavior). CFG means allowing guidance_scale>1.0 during training.

I'm not sure whether we keep single-reward model and company restriction allows that. @zhengkw18 I'd suggest reproducing the single-reward experiment according to the source code if it is urgent need. The performance plot should be quite stable.