Skip to content

[VLM] Add Qwen2VL #1553

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Jan 17, 2025
Merged

[VLM] Add Qwen2VL #1553

merged 15 commits into from
Jan 17, 2025

Conversation

yatarkan
Copy link
Contributor

Ticket: CVS-157964

@yatarkan yatarkan added this to the 2025.0 milestone Jan 15, 2025
@github-actions github-actions bot added category: visual language Visual language pipeline category: LLM LLM pipeline (stateful, static) category: GHA CI based on Github actions labels Jan 15, 2025
Wovchena
Wovchena requested changes Jan 16, 2025
View reviewed changes
yatarkan added 15 commits January 16, 2025 16:18
@yatarkan
Add qwen2_vl to vlm model type enum
a9ac262
@yatarkan
Add qwen2vl specific params to processor config
050f8f6
@yatarkan
Add image preprocessing for qwen2vl to vision encoder
80b7029
@yatarkan
Add string tokens for qwen2vl to vlm config
fadcca6
@yatarkan
Add inputs embedder for qwen2vl with utility methods, move getting po…
ec77a69 
…sition ids to class method
@yatarkan
Reuse get_position_ids class method in vlm pipeline
3185e85
@yatarkan
Add updating of 3D position ids tensor in lm_encoding
9e3be2c
@yatarkan
Calculate rope delta in inputs embedder, return it with position ids
fee5013
@yatarkan
Add optional rope delta to lm encoding
ded9d9f
@yatarkan
Propagate rope delta in vlm pipeline
f613374
@yatarkan
Add job for testing qwen2vl in vlm workflow
e4db1b9
@yatarkan
Add printing of vision embeddings merger compiled model properties
a8877c0
@yatarkan
Add qwen2vl to supported models
0ed7d1d
@yatarkan
Remove unused default values
53714b7
@yatarkan
Fix function signature
5ae6ad0
@yatarkan yatarkan requested a review from Wovchena January 16, 2025 12:31
@ilya-lavrenov ilya-lavrenov merged commit bb6138e into openvinotoolkit:master Jan 17, 2025
58 of 60 checks passed
@yatarkan yatarkan deleted the yt/qwen2vl branch January 17, 2025 09:02
@ilya-lavrenov ilya-lavrenov modified the milestones: 2025.0, 2025.1 Jan 21, 2025
@yatarkan yatarkan mentioned this pull request Jan 21, 2025
Merged
github-merge-queue bot pushed a commit that referenced this pull request Jan 23, 2025
@yatarkan
[Port] Add Qwen2VL & check not empty rendered chat template (#1590)
cd2daa2 
Port of the following PRs from master:
- #1553 
- #1574
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ilya-lavrenov ilya-lavrenov ilya-lavrenov approved these changes

@Wovchena Wovchena Wovchena approved these changes

Assignees

@Wovchena Wovchena

Labels
category: GHA CI based on Github actions category: LLM LLM pipeline (stateful, static) category: visual language Visual language pipeline no-match-files
Projects
None yet
Milestone
2025.1
Development

Successfully merging this pull request may close these issues.
3 participants
@yatarkan @ilya-lavrenov @Wovchena