Why not pad with a separate special token and ignore these pad tokens in loss?

[mishgon]

Dear LLaDA2.0 developers,

I noticed that at SFT stage you pad all sequences with <|endoftext|> token to a fixed length (2048 or 4096) which can be much larger than original sequence length (https://github.com/inclusionAI/dFactory/blob/main/tasks/dataset/data_transform.py#L136). Moreover, you include these tokens in attention_mask and compute loss for them as well. In your earlier paper about LLaDA-MoE (https://arxiv.org/abs/2509.24389) you even mention that if max_length is too large (8192), model learns to predict too many <|endoftext|> tokens which affects metrics.

My question is why don't you pad sequence with <|pad|> tokens (different from <|endoftext|>) and exclude them from attention and loss computation?

Best regards,
Mikhail

[utdawn]

Hi Mikhail,

Thanks for your great question and your interest in our work!

I'd like to clarify that the current implementation in dFactory is intended as a foundational baseline.

Furthermore, I appreciate you bringing up the comparison between LLaDA-2.0 and LLaDA-MoE, as it highlights a key architectural difference. LLaDA-2.0 is a block diffusion large language model, whereas LLaDA-MoE is a full-attention LLM. In the block diffusion framework, padding with EOS tokens primarily affects predictions within that specific block, so we believe its negative impact is less significant than in a full-attention model. You can find more details on the technical implementation of LLaDA-2.0 in our paper (https://arxiv.org/pdf/2512.15745).

That being said, your proposal to use a dedicated <|pad|> token and exclude it from loss and attention computation is excellent. In fact, the work on dLLM-Var (https://arxiv.org/pdf/2510.24605) has similar observations. These changes can definitely be implemented in dFactory with minor modifications.

Thanks again for your insightful feedback. Best regards