Transformers v5 compatibility

[Tcc0403]

This is an issue tracker for transformers v5.

Known issues

Feel free to report if you find any issue that isn't listed below.

Monkey patch

Errors when running

python3 -m pytest test/transformers/test_monkey_patch.py

See details in #960 (comment)

• AttributeError for 'language_model' in transformers v5 Fix missing property access for multimodal models #966
• KeyError: 'mrope' rope parameters for transformers v5 #1013
• Mixtral: 'MixtralDecoderLayer' object has no attribute 'block_sparse_moe'
• 'Llama4VisionConfig' object has no attribute 'rope_theta' rope parameters for transformers v5 #1013
• 'Gemma3ForConditionalGeneration' object has no attribute 'vision_tower'
• MoE breaking change, see also Moe kernel for latest transformers v5 #958

Convergence test

Errors when running

python3 -m pytest test/convergence/*

• Some XXXTokenizerFast are deprecated: No module named transformers.models.gemma.tokenization_gemma_fast Fix/transformers v5 gemma tokenizer #1030
• llava: vision_tower generate outputs.hdiden_states=None Llava vision_tower monkey patch generates hidden_states=None #1011
• qwen2_vl, qwen2_5_vl: 'Qwen2_5_VLConfig' object has no attribute 'hidden_size' Qwen2VLConfig and Qwen2_5_VLConfig have no attribute hidden_size #1012

Benchmarking

• qwen2vl_mrope: Qwen2VLRotaryEmbedding expects config argument instead of int(hidden_dim) Benchmark benchmark_qwen2vl_mrope.py fails with new transformers #1015

For Contributors

Environment

Install the latest release candidate in your local environment. See #994 for the release candidate version.

uv pip install transformers==5.0.0rc3

Check transformers version

uv pip show transformers

Pull Requests

When opening a pull request, choose https://github.com/linkedin/Liger-Kernel/tree/transformers-5.0.0rc1 as your base branch.