Add V-JEPA 2 #38746

qubvel · 2025-06-11T08:59:56Z

What does this PR do?

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

…el-addition-jepa2 into koustuvs/oss

src/transformers/models/vjepa2/configuration_vjepa2.py

docs/source/en/model_doc/vjepa2.md

pcuenca · 2025-06-11T09:15:33Z

docs/source/en/model_doc/vjepa2.md

+from torchcodec.decoders import VideoDecoder
+import numpy as np
+
+processor = AutoVideoProcessor.from_pretrained("facebook/vjepa2-vitl-fpc64-256")


Just flagging to confirm this be the final repo name @merveenoyan @ariG23498 ?

yes this is the final repo name!

src/transformers/models/vjepa2/convert_vjepa2_to_hf.py

pcuenca · 2025-06-11T09:22:26Z

src/transformers/models/vjepa2/convert_vjepa2_to_hf.py

+    return image
+
+
+def upload_original_ckpts(model_name):


Do we need to expose this in the final script? If not, we can remove S3_MODELS above too (and the --upload_original arg below)

I have no strong opinion, but we can keep it for a while because the model is not in the final state yet and might want to update checkpoints or upload new ones, e.g. classification models

HuggingFaceDocBuilderDev · 2025-06-11T09:23:47Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

…into koustuvs/oss

LysandreJik

good, typos should be removed from the library

* adding model and conversion scripts * add imports to test vjepa conversion * fix imports and make conversion work * fix computation for short side * replace attention with library attention function * cleanup more attention classes * remove config overrides * add test cases, fix some of the failing ones * fix the model outputs * fix outputs of the model per review * fix too big model test case * fix styling __init__.py * fix initialization test * remove all asserts per review * update sorting unsorting logic as per feedback * remove is_video per review * remove another is_video segment * remove unwanted stuff * small fixes * add docstrings for the model * revert adding vjepa2 config here * update styling * add config docstrings (wip) * fix dpr issue * removed test failing issues * update styles * merge predictor configs into main config * remove processing code, add video processor * remove permute which is not necessary now * fix styles * updated vjepa2 to be in video_processing_auto * update comment for preprocessing * test integration test and fix the outputs * update test values, change test to look at repeated frames for a given image * add a simple video processing test * refactoring pixel_values_videos and upload ckpts to original * fix torch_fx test cases * remove unused config * add all config docstrings * add more integration tests * add basic doc * revert unwanted styling changes * working make fixup * Fix model_type in config * update attention implementation to fit new hf standards * fix the preprocessing logic, ensure it matches the original model * remove use_rope logic, cleanup * fix docstrings * Further cleanup, update doc * Fix model prefix * fix get_vision_features * VJEPA2Embeddings style refactor * nit, style comment * change modules default values * Only `str` activation in config * GradientCheckpointingLayer * fixup * fix conversion script * Remove return_dict * remove None return typehint * Refactor VJEPA2Layer, remove use_SiLU * Fix fx tests * dpr -> drop_path_rates * move *ModelOutput on top * format docs bit * update docs * update docs * update doc example * remove prune_heads from model * remove unused config params * refactor embed signature * Add vjepa to docs * Fix config docstring * update defaults * Update docs/source/en/model_doc/vjepa2.md Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * Update docs/source/en/model_doc/vjepa2.md Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * Fix import * Min refactoring * Update HUB_SOURCE and HUB_REPO in conversion script * Add missing headers * VJEPA -> V-JEPA in docs * Add image to doc * fix style * fix init weights * change checkpoint name in modeling tests --------- Co-authored-by: Koustuv Sinha <koustuv.sinha@mail.mcgill.ca> Co-authored-by: yonigozlan <yoni.gozlan@huggingface.co> Co-authored-by: Yoni Gozlan <74535834+yonigozlan@users.noreply.github.com> Co-authored-by: Koustuv Sinha <koustuvsinha@gmail.com> Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

koustuvsinha and others added 30 commits June 3, 2025 17:05

adding model and conversion scripts

6c1c472

add imports to test vjepa conversion

103bde8

fix imports and make conversion work

d09c1c0

fix computation for short side

3fd6bf2

replace attention with library attention function

aae79f7

cleanup more attention classes

41957a6

remove config overrides

f46df32

add test cases, fix some of the failing ones

174ba39

fix the model outputs

3e41279

fix outputs of the model per review

8711628

fix too big model test case

da7f76a

Merge remote-tracking branch 'upstream/main' into koustuvs/oss

beba328

Merge branch 'koustuvs/oss' of https://github.com/huggingface/new-mod…

d54b0a4

…el-addition-jepa2 into koustuvs/oss

fix styling __init__.py

db6b5e3

fix initialization test

df77afe

remove all asserts per review

239b30f

update sorting unsorting logic as per feedback

dd4850d

remove is_video per review

d8b18b1

remove another is_video segment

eb955bf

remove unwanted stuff

a382ebb

small fixes

de97de0

add docstrings for the model

30f6feb

revert adding vjepa2 config here

f5a07b2

update styling

d90da4d

add config docstrings (wip)

238f8f3

fix dpr issue

9bc533d

removed test failing issues

0354e0c

update styles

fdb6697

merge predictor configs into main config

e203d41

remove processing code, add video processor

3c60022

yonigozlan and others added 2 commits June 11, 2025 03:12

remove unused config params

918d302

refactor embed signature

4275757

qubvel commented Jun 11, 2025

View reviewed changes

src/transformers/models/vjepa2/configuration_vjepa2.py Outdated Show resolved Hide resolved

qubvel added 2 commits June 11, 2025 09:10

Add vjepa to docs

ddd1c36

Fix config docstring

b8f67a4

pcuenca reviewed Jun 11, 2025

View reviewed changes

docs/source/en/model_doc/vjepa2.md Outdated Show resolved Hide resolved

pcuenca reviewed Jun 11, 2025

View reviewed changes

docs/source/en/model_doc/vjepa2.md Outdated Show resolved Hide resolved

pcuenca reviewed Jun 11, 2025

View reviewed changes

src/transformers/models/vjepa2/convert_vjepa2_to_hf.py Outdated Show resolved Hide resolved

pcuenca reviewed Jun 11, 2025

View reviewed changes

qubvel and others added 4 commits June 11, 2025 11:07

update defaults

6d6aadc

Update docs/source/en/model_doc/vjepa2.md

9661409

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

Update docs/source/en/model_doc/vjepa2.md

f67a4d7

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

Fix import

2908c5c

qubvel marked this pull request as ready for review June 11, 2025 11:43

qubvel and others added 9 commits June 11, 2025 12:47

Merge branch 'main' into koustuvs/oss

91028fc

Min refactoring

fb95d52

Update HUB_SOURCE and HUB_REPO in conversion script

39aafc3

Add missing headers

7e660ef

VJEPA -> V-JEPA in docs

9061b3a

Add image to doc

284b3e5

fix style

a7d750b

Merge branch 'koustuvs/oss' of https://github.com/qubvel/transformers …

f80ff37

…into koustuvs/oss

fix init weights

1058246

LysandreJik approved these changes Jun 11, 2025

View reviewed changes

change checkpoint name in modeling tests

a85656e

qubvel changed the title ~~Fix typo in docs~~ Add V-JEPA 2 Jun 11, 2025

qubvel merged commit 84710a4 into huggingface:main Jun 11, 2025
20 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add V-JEPA 2 #38746

Add V-JEPA 2 #38746

Uh oh!

qubvel commented Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pcuenca Jun 11, 2025

Uh oh!

koustuvsinha Jun 11, 2025

Uh oh!

Uh oh!

pcuenca Jun 11, 2025

Uh oh!

qubvel Jun 11, 2025

Uh oh!

HuggingFaceDocBuilderDev commented Jun 11, 2025

Uh oh!

LysandreJik left a comment

Uh oh!

Uh oh!

Uh oh!

Add V-JEPA 2 #38746

Add V-JEPA 2 #38746

Uh oh!

Conversation

qubvel commented Jun 11, 2025

What does this PR do?

Before submitting

Who can review?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pcuenca Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

koustuvsinha Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pcuenca Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

qubvel Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Jun 11, 2025

Uh oh!

LysandreJik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!