Simple Demo

[Zeqing-Wang]

Hey! Thanks for your great work. Do u have any plan to provide a simple demo, i.e., input a video and a question, not a benchmark?

[Zeqing-Wang]

Additionally, will you release the scripts for zero-shot QA eval? Tks~

[Yogesh914]

Hey, I have written some code for a simple demo on google colab if you like: https://github.com/Yogesh914/VideoGPT-plus/blob/main/VideoGPT%2B_Demo.ipynb

[YoungjaeDev]

@Yogesh914

Is it possible to take real-time camera input and reference it? How long does it take to reference?

[Lynn-798]

嘿，如果您愿意，我已经为 google colab 上的一个简单演示编写了一些代码：https://github.com/Yogesh914/VideoGPT-plus/blob/main/VideoGPT%2B_Demo.ipynb

thank you very much

[SerenWhite]

Hey, I have written some code for a simple demo on google colab if you like: https://github.com/Yogesh914/VideoGPT-plus/blob/main/VideoGPT%2B_Demo.ipynb

Thank you very much, but I have a question.：the warning indicates that some model weights from the checkpoint are either missing or don't match the current model structure. For example, certain weights were newly initialized while others were not found in the checkpoint. Is this expected behavior, or should I take further action to resolve these discrepancies?
For example:
......
Loading additional VideoGPT+ weights...
Loading LoRA weights...
Merging LoRA weights...
Model is loaded...
load_state_dict: _IncompatibleKeys(missing_keys=[], unexpected_keys=['temp', 'text_encoder.bert.embeddings.position_ids', ......]
......
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00, 1.39s/it]
Some weights of VideoGPTPlusPhi3ForCausalLM were not initialized from the model checkpoint at /ceph/ssd4/pengj25/VideoGPT-plus/Phi-3-mini-4k-instruct and are newly initialized: ['model.image_mm_projector.0.bias', 'model.image_mm_projector.0.weight'......]