Openvision3 usage

[stellaludai]

Hi authors,
can you provide the code snippet for using openvision 3? I encountered the following bugs:

import torch
import torch.nn.functional as F
from urllib.request import urlopen
from PIL import Image
from open_clip import create_model_from_pretrained, get_tokenizer
model, preprocess = create_model_from_pretrained(f"hf-hub:{hf_repo}
")
open_clip_pytorch_model.bin: 100%|█| 1.21G/1.21G [12:58<00:00, 1.
Traceback (most recent call last):
File "", line 1, in
File "/opt/conda/envs/openvision/lib/python3.10/site-packages/open_clip/f
actory.py", line 1062, in create_model_from_pretrained
model = create_model(
File "/opt/conda/envs/openvision/lib/python3.10/site-packages/open_clip/f
actory.py", line 501, in create_model
model = model_class(**final_model_cfg, cast_dtype=cast_dtype)
File "/opt/conda/envs/openvision/lib/python3.10/site-packages/open_clip/m
odel.py", line 283, in init
self.visual = _build_vision_tower(embed_dim, vision_cfg, quick_gelu
, cast_dtype)
File "/opt/conda/envs/openvision/lib/python3.10/site-packages/open_clip/m
odel.py", line 140, in _build_vision_tower
vision_cfg = CLIPVisionCfg(**vision_cfg)
TypeError: CLIPVisionCfg.init() got an unexpected keyword argument
'in_channels'

[Letian2003]

Hi,

We have provided a script openvision3_test.py for OpenVision 3.

This script uses an updated version of open_clip (src.convert_upload.open_clip) which has the keyword 'in_channels' and this solves your error.

Please let us know is there's any problem when using OpenVision 3.

[stellaludai]

thanks but still need some help:) This script works for encoding and feature extraction, as well as decoding from flux. But here is a continuing question: how to decode the image from your extracted features after flux + vit? @Letian2003