Fix missing initializations for models created in 2023 by bvantuan · Pull Request #39239 · huggingface/transformers

bvantuan · 2025-07-06T15:14:13Z

What does this PR do?

Fixes missing weight initializations for models created in 2023.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@Cyrilvallez

ArthurZucker · 2025-07-07T13:04:15Z

cc @Cyrilvallez as we are also working on putting the default in the PreTrainedModel

Cyrilvallez · 2025-07-07T13:08:43Z

Super happy to see you're following again on a new batch of models @bvantuan! 🚀🤗 Let me know when this is ready!

bvantuan · 2025-07-07T13:37:50Z

Yes, of course! Really excited to keep contributing whenever I have time.

bvantuan · 2025-07-09T10:47:37Z

src/transformers/models/clap/modeling_clap.py

            module.weight.data.normal_(mean=0.0, std=factor * 0.02)
-
-        elif isinstance(module, nn.LayerNorm):
+        elif isinstance(module, (nn.LayerNorm, nn.BatchNorm2d)):


https://github.com/LAION-AI/CLAP/blob/1fd4c37df5ffbfcfbad5415c170bc66cf94c9994/src/laion_clap/clap_module/pann_model.py#L26-L29

bvantuan · 2025-07-09T10:54:38Z

src/transformers/models/rwkv/modeling_rwkv.py

+        elif isinstance(module, nn.Linear):
+            shape = module.weight.data.shape
+            gain = 1.0
+            scale = 1.0  # extra scale for gain
+            if module.bias is not None:
+                module.bias.data.zero_()
+            if shape[0] > shape[1]:
+                gain = math.sqrt(shape[0] / shape[1])
+            if shape[0] == self.config.vocab_size and shape[1] == self.config.hidden_size:  # final projection?
+                scale = 0.5
+
+            gain *= scale
+            nn.init.orthogonal_(module.weight, gain=gain)
+        elif isinstance(module, nn.Embedding):
+            shape = module.weight.data.shape
+            gain = 1e-4 * math.sqrt(max(shape[0], shape[1]))
+            nn.init.orthogonal_(module.weight, gain=gain)


https://github.com/BlinkDL/RWKV-LM/blob/cc2ee37c04422e37d1b022fbae60995bb4eebc2f/RWKV-v3/src/model.py#L62-L105

bvantuan · 2025-07-09T11:08:26Z

Cc @Cyrilvallez ! The PR is now ready and awaiting your review😊.

Cyrilvallez

All right, nice work! 🤗 Very happy to see this! Just a few comments to simplify!
Especially, the pattern copy_(torch.randn(...)) is an anti-pattern: we should use normal_ instead directly, which makes it much simpler and we don't have to worry about the shapes

src/transformers/models/bridgetower/modeling_bridgetower.py

src/transformers/models/kosmos2/modeling_kosmos2.py

src/transformers/models/mgp_str/modeling_mgp_str.py

src/transformers/models/patchtst/modeling_patchtst.py

src/transformers/models/sam_hq/modeling_sam_hq.py

src/transformers/models/seamless_m4t_v2/modeling_seamless_m4t_v2.py

src/transformers/models/swiftformer/modeling_swiftformer.py

src/transformers/models/tvp/modeling_tvp.py

src/transformers/models/vits/modeling_vits.py

bvantuan · 2025-07-15T09:23:50Z

Thanks a lot for reviewing, @Cyrilvallez ! I’m really glad you’re happy with it. I completely agree—using normal_ instead of copy_(torch.randn(...)) is much cleaner and more elegant. I’ve simplified that part and addressed the other comments as well.

github-actions · 2025-07-21T12:33:19Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: align, autoformer, bridgetower, bros, clap, clvp, efficientnet, fastspeech2_conformer, informer, kosmos2, mgp_str, mobilevit, mobilevitv2, mra, nllb_moe, omdet_turbo

Cyrilvallez

Alright, thanks a lot for this new batch! Always greatly appreciated! 🤗

…9239) * fix SwiftFormer * fix Kosmos2 * fix Owlv2 * fix Sam * fix Vits * fix Pvt * fix MobileViTV2 * fix PatchTST * fix Bros * fix Informer * fix BridgeTower * fix Mra and Yoso * fix Rwkv * fix EfficientNet * fix NllbMoe * fix Tvp * fix Clap * fix Autoformer * fix SwiftFormer * fix Mgpstr * fix Align * fix VitMatte * fix SpeechT5 * add conditional check for parameters * fix SpeechT5 * fix TimmBackbone and Clvp * fix SwiftFormer * fix SeamlessM4T and SeamlessM4Tv2 * fix Align * fix Owlv2 and OwlViT * add reviewed changes * add reviewed changes * fix typo --------- Co-authored-by: Cyril Vallez <cyril.vallez@huggingface.co>

ydshieh · 2025-07-25T14:23:45Z

Hi @bvantuan

Thanks for the PR. Could you check

    "models_fastspeech2_conformer": {
        "job_link": "https://github.com/huggingface/transformers/actions/runs/16433389284/job/46439076372",
        "single": [
            {
                "line": "tests/models/fastspeech2_conformer/test_modeling_fastspeech2_conformer.py::FastSpeech2ConformerModelTest::test_can_init_all_missing_weights",
                "trace": "(line 687)  AssertionError: False is not true : The following keys are not properly handled by `_init_weights()`:"
            },
            {
                "line": "tests/models/fastspeech2_conformer/test_modeling_fastspeech2_conformer.py::FastSpeech2ConformerWithHifiGanTest::test_can_init_all_missing_weights",
                "trace": "(line 687)  AssertionError: False is not true : The following keys are not properly handled by `_init_weights()`:"
            }
        ]
    },

which are skipped before this PR but now failing since this PR?

bvantuan · 2025-07-26T02:55:28Z

Hi @ydshieh @Cyrilvallez ! I think the FastSpeech2Conformer tests require @require_torch_accelerator, which might explain why they didn’t run on this CI. I opened a PR #39689 to fix this.

Cyrilvallez · 2025-07-28T08:43:59Z

Indeed @bvantuan, the tests should not have that decorator as it prevents them being run by the CI! Thanks @ydshieh for raising the issue, the PR linked by @bvantuan will fix it! 🤗 I'm reactivating the tests as well!

…9239) * fix SwiftFormer * fix Kosmos2 * fix Owlv2 * fix Sam * fix Vits * fix Pvt * fix MobileViTV2 * fix PatchTST * fix Bros * fix Informer * fix BridgeTower * fix Mra and Yoso * fix Rwkv * fix EfficientNet * fix NllbMoe * fix Tvp * fix Clap * fix Autoformer * fix SwiftFormer * fix Mgpstr * fix Align * fix VitMatte * fix SpeechT5 * add conditional check for parameters * fix SpeechT5 * fix TimmBackbone and Clvp * fix SwiftFormer * fix SeamlessM4T and SeamlessM4Tv2 * fix Align * fix Owlv2 and OwlViT * add reviewed changes * add reviewed changes * fix typo --------- Co-authored-by: Cyril Vallez <cyril.vallez@huggingface.co>

fix SwiftFormer

1643a12

bvantuan marked this pull request as draft July 6, 2025 15:14

bvantuan added 9 commits July 7, 2025 06:15

fix Kosmos2

92602b2

fix Owlv2

acde90a

fix Sam

a29aa01

fix Vits

20f3924

fix Pvt

740a318

fix MobileViTV2

5875051

fix PatchTST

9bce1db

fix Bros

3056581

fix Informer

be67ca0

bvantuan added 16 commits July 7, 2025 16:04

fix BridgeTower

4865885

fix Mra and Yoso

23f69a7

fix Rwkv

9fb70cb

fix EfficientNet

bbac997

fix NllbMoe

a6aaca9

fix Tvp

c787598

fix Clap

378d44a

fix Autoformer

14abd06

fix SwiftFormer

60697e9

fix Mgpstr

45c03f7

fix Align

b44cda7

fix VitMatte

7ad83b2

fix SpeechT5

c98eb21

add conditional check for parameters

8822f26

fix SpeechT5

7cb069c

fix TimmBackbone and Clvp

2af932a

bvantuan commented Jul 9, 2025

View reviewed changes

bvantuan marked this pull request as ready for review July 9, 2025 11:08

Cyrilvallez reviewed Jul 14, 2025

View reviewed changes

bvantuan and others added 4 commits July 15, 2025 10:21

add reviewed changes

c2403f0

Merge branch 'main' into fix-missing-initializations-2023

b973f33

add reviewed changes

4618da2

fix typo

d14b492

bvantuan mentioned this pull request Jul 15, 2025

Fix missing initializations for models created in 2024 #38987

Merged

bvantuan and others added 4 commits July 17, 2025 12:12

Merge branch 'main' into fix-missing-initializations-2023

6d49420

Merge branch 'main' into fix-missing-initializations-2023

5622773

Merge branch 'main' into fix-missing-initializations-2023

9a31452

Merge branch 'main' into fix-missing-initializations-2023

3c9c09f

Cyrilvallez approved these changes Jul 21, 2025

View reviewed changes

Cyrilvallez merged commit 6b3a1f2 into huggingface:main Jul 21, 2025
23 checks passed

bvantuan mentioned this pull request Jul 26, 2025

Fix missing initialization of FastSpeech2Conformer #39689

Merged

Conversation

bvantuan commented Jul 6, 2025

What does this PR do?

Who can review?

Uh oh!

ArthurZucker commented Jul 7, 2025

Uh oh!

Cyrilvallez commented Jul 7, 2025

Uh oh!

bvantuan commented Jul 7, 2025

Uh oh!

bvantuan Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

bvantuan Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

bvantuan commented Jul 9, 2025

Uh oh!

Cyrilvallez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bvantuan commented Jul 15, 2025

Uh oh!

github-actions bot commented Jul 21, 2025

Uh oh!

Cyrilvallez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ydshieh commented Jul 25, 2025

Uh oh!

bvantuan commented Jul 26, 2025

Uh oh!

Cyrilvallez commented Jul 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants