🏔️ REACT로 보여주는 KoLLaVA

[Dataset] [Model] [Paper Review]

Korean Large Language and Vision Assistant (feat. LLaVA)
이미지 기반 한국어 대화 가능한 멀티모달 모델

KoLLaVA Logo

Update Logs

23.07.01
- 💥 🤗 KoLLaVA-KULLM-13B-8bit 공개 : KULLM을 KoLLaVA-Instruct-150k으로 학습
- 💥 클라우드 GPU 대여 비용 문제로 데모를 일시 중지합니다🥲
23.06.24
- 💥 🤗 Ko-Otter-9B-LACR-v0 공개 : Otter를 KoLLaVA_Complex_Resoning_77k 데이터셋으로 학습
23.06.18
- 💥 ~~Gradio를 이용한 데모를 오픈합니다! (RTX 3090 GPU 1개)~~
23.06.12
- 💥 한국어 Visual Instruction 데이터셋으로 학습한 🤗KoLLaVA-KoVicuna-7B 공개
- 💥 Colab(Pro) 이용한 inference 예시
23.06.09
- 💥 LLaVA의 학습 데이터셋을 한국어로 번역한 🤗KoLLaVA-Instruct-150K / KoLLaVA-CC3M-Pretrain-595K 공개

Visual Chat Example

Visual Instruction Dataset

🤗 KoLLaVA-Instruct-150K : LLaVA의 instruction-following 데이터셋을 DeepL로 번역

English	Korean
llava_instruct_150k.json	ko_llava_instruct_150k.json
conversation_58k.json	ko_conversation_58k.json
detail_23k.json	ko_detail_23k.json
complex_reasoning_77k.json	ko_complex_reasoning_77k.json

Details

Visual instruction tuning에 사용하는 instruction-following 데이터는 GPT-4로 생성된 데이터입니다. 이때 GPT-4의 인풋으로는 텍스트만 넣어줍니다(이미지 X). 구체적으로는 image-text pair 데이터셋인 COCO의 텍스트 정보(caption, bounding box)만을 이용해 instruction-following 형식의 데이터를 생성한 것입니다. 이러한 데이터 생성 파이프라인이 궁금하신 분은 블로그를 참고해주세요.

License: Attribution-NonCommercial 4.0 International | OpenAI policy 준수

Image Dataset

Finetuning에 사용되는 이미지 데이터셋은 COCO-train2014 입니다.

wget http://images.cocodataset.org/zips/train2014.zip

Pretraining Dataset

🤗 KoLLaVA-CC3M-Pretrain-595K : LLaVA Pretrain 데이터셋의 index에 맞춰 Ko-CC3M 한국어 caption 추출

Data	English	Korean	Size
CC3M Concept-balanced 595K	chat.json	ko_chat.json	211 MB / 229 MB

Details

사전학습 데이터셋은 CC3M을 필터링해 생성했으며, 595K개의 데이터로 이루어져 있습니다. 데이터셋 구조와 영어 버전 다운로드 방법에 대한 자세한 설명은 여기를, 한국어 데이터셋은 여기를 참고하세요. (주의 : DeepL로 번역한 결과가 아니며, 품질이 조금 떨어질 수 있습니다.)

License: CC-3M 준수

Image Dataset

images.zip - LLaVA의 저자들은 사전학습에 사용한 이미지 파일도 공유했습니다. 이 이미지 파일은 연구 외에 다른 용도로 사용해서는 안 되며, 이미지의 사용은 CC3M의 라이선스를 준수해야 합니다. 원본 CC3M 데이터셋 소유자 혹은 참조된 이미지의 소유자가 요청할 경우 언제든지 해당 이미지는 삭제될 수 있습니다.

Install

Clone 후 해당 디렉토리로 이동

 git clone https://github.com/tabtoyou/KoLLaVA.git

Package 설치

 conda create -n kollava python=3.10 -y
 conda activate kollava
 pip install --upgrade pip 
 pip install -e .

학습 진행할 경우 추가 Package 설치

pip install ninja
pip install flash-attn==1.0.2

Inference

아래 명령어로는 single-turn 대화만 가능합니다. 파라미터(temperature, max length)를 조절해 다양한 출력 결과를 얻을 수 있습니다. 현재 7B 모델은 1epoch만 finetune한 모델로 아직 부족한 부분이 많습니다. 추후 13B 모델, Quantized 모델 등 개선된 모델들을 공개할 예정입니다. (Model License: Apache License 2.0)

python -m llava.eval.run_llava \
    --model-name tabtoyou/KoLLaVA-KoVicuna-7b \
    --image-file /path/to/image \
    --query "이 이미지에 대해 자세히 설명하세요."

결과 예시

이미지 :

질문 :

이미지 속 재료들로 어떤 음식을 만들 수 있을까요?

답변 :

case1)

이미지에는 냉장고 문이 열려 있고 내부에 다양한 식품이 가득 차 있습니다. 냉장고에는 블루베리, 딸기, 당근, 우유, 버터, 코코넛 및 기타 과일과 채소가 들어 있습니다. 이러한 재료를 사용하여 맛있는 간식이나 식사를 만들 수 있습니다. 예를 들어 블루베리와 딸기를 섞어 블루베리 딸기 샐러드를 만들 수 있습니다. 또는 당근과 우유를 섞어 당근 샐러드를 만들거나 당근을 버터와 함께 구워 따뜻하게 먹을 수 있습니다. 또한 버터와 코코넛을 사용하여 코코넛 버터 파르페를 만들 수도 있습니다. 이러한 재료를 사용하여 창의력을 발휘하여 맛있는 간식이나 식사를 만들 수 있습니다.

case2)

이미지에는 다양한 식품이 있는 냉장고가 있습니다. 냉장고에는 과일, 빵, 우유, 물, 주스, 달걀, 치즈, 버터, 베리류, 블루베리, 딸기, 당근 등 다양한 식품이 들어 있습니다. 이러한 재료를 사용하여 다양한 요리를 만들 수 있습니다. 몇 가지 가능한 옵션은 다음과 같습니다:

과일 샐러드: 베리류, 딸기, 당근을 섞어 신선하고 건강한 샐러드를 만들 수 있습니다.

과일 베이스 스무디: 베리류, 딸기, 당근을 섞어 상큼한 스무디를 만들어 보세요.

과일 샌드위치: 베리류, 딸기, 당근을 샌드위치 빵에 얹어 맛있는 샌드위치를 만들어 보세요.

과일 디저트: 베리류, 딸기, 당근을 섞어 과일 코블러나 팬케이크와 같은 달콤한 디저트를 만들어 보세요.

과일 기반 요리: 베리류, 딸기, 당근을 샐러드나 수프에 넣어 맛있는 요리를 만들어 보세요.

이는 몇 가지 예에 불과하며, 냉장고에 있는 다양한 식품을 사용하여 다양한 요리를 만들 수 있는 가능성은 무궁무진합니다.

Training

클라우드 GPU 대여 서비스인 vast.ai를 이용해 학습을 진행했습니다. 4개의 A100(80GB) GPU를 대여했으며 Disk Space는 200GB 이상을 추천드립니다(시간 당 약 $7.44). 인스턴스 생성 시 Docker image로 pytorch/pytorch:2.0.1-cuda11.7-cudnn8-devel 를 사용했습니다.

Pretrain

./scripts/pretrain_lightning.sh

#!/bin/bash

WEIGHT_VERSION=$1

# Pretraining (5~6 hours on 4 A100/80GB GPU)
torchrun --nnodes=1 --nproc_per_node=4 --master_port=25001 \
   llava/train/train_mem.py \
   --model_name_or_path junelee/ko_vicuna_7b \
   --version $WEIGHT_VERSION \
   --data_path /path/to/ko_chat.json \
   --image_folder /path/to/CC3M_images \
   --vision_tower openai/clip-vit-large-patch14 \
   --tune_mm_mlp_adapter True \
   --mm_vision_select_layer -2 \
   --mm_use_im_start_end \
   --bf16 True \
   --output_dir ./checkpoints/kollava-lightning-7b-pretrain \
   --num_train_epochs 1 \
   --per_device_train_batch_size 32 \
   --per_device_eval_batch_size 4 \
   --gradient_accumulation_steps 1 \
   --evaluation_strategy "no" \
   --save_strategy "steps" \
   --save_steps 2400 \
   --save_total_limit 1 \
   --learning_rate 2e-3 \
   --weight_decay 0. \
   --warmup_ratio 0.03 \
   --lr_scheduler_type "cosine" \
   --logging_steps 1 \
   --tf32 True \
   --model_max_length 2048 \
   --gradient_checkpointing True \
   --dataloader_num_workers 4 \
   --lazy_preprocess True \
   --report_to wandb

# Extract projector features
python scripts/extract_mm_projector.py \
 --model_name_or_path ./checkpoints/kollava-lightning-7b-pretrain \
 --output ./checkpoints/mm_projector/kollava-lightning-7b-pretrain.bin

Run

sh scripts/pretrain_lightning.sh v0

Visual instruction tuning (Finetune)

./scripts/finetune_lightning.sh

#!/bin/bash

WEIGHT_VERSION=$1

# Visual instruction tuning (6~7 hour on 4 A100/80GB GPU)
torchrun --nnodes=1 --nproc_per_node=4 --master_port=25001 \
    llava/train/train_mem.py \
    --model_name_or_path junelee/ko_vicuna_7b \
    --version $WEIGHT_VERSION \
    --data_path /path/to/ko_llava_instruct_150k.json \
    --image_folder /path/to/coco/train2014 \
    --vision_tower openai/clip-vit-large-patch14 \
    --pretrain_mm_mlp_adapter ./checkpoints/mm_projector/kollava-lightning-7b-pretrain.bin \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end True \
    --bf16 True \
    --output_dir ./checkpoints/kollava-lightning-7b-finetune \
    --num_train_epochs 1 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 1500 \
    --save_total_limit 1 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --dataloader_num_workers 4 \
    --lazy_preprocess True \
    --report_to wandb

Run

sh scripts/finetune_lightning.sh v0

Serving

Web UI 데모 실행 방법

여러 터미널에서 아래 명령을 병렬적으로 실행해야 합니다. Linux의 경우 tmux/screen과 같은 terminal multiplexer를 이용해, 아래 명령어를 각각 다른 터미널 세션에서 순차적으로 실행해주세요.

Launch a controller

python -m llava.serve.controller --host 0.0.0.0 --port 10000

Launch a model worker

python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path tabtoyou/KoLLaVA-KoVicuna-7b --multi-modal

모델 로드를 완료하고 "Uvicorn running on ..."이 표시될 때까지 기다립니다. 멀티 GPU로 load 할 경우 --num-gpus 2 로 사용할 gpu 개수를 입력해줍니다.

Launch a gradio web server.

python -m llava.serve.gradio_web_server --controller http://localhost:10000 --share

https://33166c79a780850dad.gradio.live 와 같은 데모 link가 출력됩니다.

To-do

Finetuning 데이터셋 한국어 번역 (LLaVA-Instruct-150K)
Pretraining 데이터셋 한국어 번역 (LLaVA-CC3M-Pretrain-595K)
LLaVA 모델에서 Vicuna -> KoVicuna-7B 대체 후 학습
Ko-Otter 모델 학습 및 허깅페이스 공개
KoLLaVA-13B 모델 학습 및 허깅페이스 공개
QLoRA 이용해 low GPU memory에서도 학습할 수 있도록 (RTX 3090 등)
KoLLaVA의 linear layer를 Q-former로 업데이트(InstructBLIP)

Team

KoLLaVA 프로젝트는 딥러닝 스터디 구성원들과 함께 진행했습니다.

팀원 : Jeonghyeon, Seongyeon, Seonghwan, Seungwoo, Seonghun, Taebaek

🌋 LLaVA: Large Language and Vision Assistant

Visual instruction tuning towards large language and vision models with GPT-4 level capabilities.

[Project Page] [Paper] [Demo] [Data] [Model]

Visual Instruction Tuning
Haotian Liu*, Chunyuan Li*, Qingyang Wu, Yong Jae Lee (*Equal Contribution)

Usage and License Notices: The data, code and checkpoint is intended and licensed for research use only. They are also restricted to uses that follow the license agreement of LLaMA, Vicuna and GPT-4. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.

Data Download

Data file name	Size
llava_instruct_150k.json	229 MB
llava_instruct_80k.json	229 MB
conversation_58k.json	126 MB
detail_23k.json	20.5 MB
complex_reasoning_77k.json	79.6 MB

To download our langauge-image multimodal instruction-folllowing dataset LLaVA-Instruct-150K, please run the following script:

sh download_data.sh

Pretraining Dataset

The pretraining dataset used in this release is a subset of CC-3M dataset, filtered with a more balanced concept coverage distribution. Please see here for a detailed description on the dataset structure and how to download the images.

If you already have CC-3M dataset on your disk, the image names follow this format: GCC_train_000000000.jpg. You may edit the image field correspondingly if necessary.

Data	Chat File	Meta Data	Size
CC-3M Concept-balanced 595K	chat.json	metadata.json	211 MB
LAION/CC/SBU BLIP-Caption Concept-balanced 558K	blip_laion_cc_sbu_558k.json	metadata.json	181 MB

GPT-4 Prompts

We provide our prompts and few-shot samples for GPT-4 queries, to better facilitate research in this domain. Please check out the prompts folder for three kinds of questions: conversation, detail description, and complex reasoning.

They are organized in a format of system_message.txt for system message, pairs of abc_caps.txt for few-shot sample user input, and abc_conv.txt for few-shot sample reference output.

Note that you may find them in different format. For example, conversation is in jsonl, and detail description is answer-only. The selected format in our preliminary experiments work slightly better than a limited set of alternatives that we tried: jsonl, more natural format, answer-only. If interested, you may try other variants or conduct more careful study in this. Contributions are welcomed!

Install

Clone this repository and navigate to LLaVA folder

git clone https://github.com/haotian-liu/LLaVA.git
cd LLaVA

Install Package

conda create -n llava python=3.10 -y
conda activate llava
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

NOTE: [Update 4/30/23] We have successfully moved LLaVA framework to this repo, without the need of a special transformers modified by us. If you install our repo before 4/30/23, please reinstall transformers following the instructions here.

Install additional packages for training cases

pip install ninja
pip install flash-attn==1.0.2

Upgrade to v0.1

NOTE: If you install our package before 4/30/23, please make sure to execute the command below to correctly upgrade to v0.1. You may try a clean install as well.

git pull
pip uninstall transformers
pip install git+https://github.com/huggingface/transformers@cae78c46
pip install -e .

LLaVA Weights

We release LLaVA weights as delta weights to comply with the LLaMA model license. You can add our delta to the original LLaMA weights to obtain the LLaVA weights.

Instructions:

Get the original LLaMA weights in the huggingface format by following the instructions here.
Use the following scripts to get LLaVA weights by applying our delta (13b-v0, 7b-v0, lightning-7B-v1-1). It will automatically download delta weights from our Hugging Face account.

LLaVA-13B

This conversion command needs around 60 GB of CPU RAM.

python3 -m llava.model.apply_delta \
    --base /path/to/llama-13b \
    --target /output/path/to/LLaVA-13B-v0 \
    --delta liuhaotian/LLaVA-13b-delta-v0

LLaVA-7B

This conversion command needs around 30 GB of CPU RAM.

python3 -m llava.model.apply_delta \
    --base /path/to/llama-7b \
    --target /output/path/to/LLaVA-7B-v0 \
    --delta liuhaotian/LLaVA-7b-delta-v0

LLaVA pretrained projector weights

The initial release is pretrained on LLaVA-filtered CC3M 595K with 1 epoch. The pretrained weights are released here.

You may perform instruction tuning on our pretrained checkpoints, by using our visual instruction tuning data following the instructions here.

Serving

Web UI

Launch a controller

python -m llava.serve.controller --host 0.0.0.0 --port 10000

Launch a model worker

python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path ./checkpoints/LLaVA-13B-v0 --multi-modal

Wait until the process finishes loading the model and you see "Uvicorn running on ...".

Launch a model worker (Multiple GPUs, when GPU VRAM <= 24GB)

If your the VRAM of your GPU is less than 24GB (e.g., RTX 3090, RTX 4090, etc.), you may try running it with multiple GPUs.

python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path ./checkpoints/LLaVA-13B-v0 --multi-modal --num-gpus 2

Wait until the process finishes loading the model and you see "Uvicorn running on ...".

Launch a gradio web server.

python -m llava.serve.gradio_web_server --controller http://localhost:10000

You can open your browser and chat with a model now.

CLI Inference

A starting script for inference with LLaVA without the need of Gradio interface. The current implementation only supports for a single-turn Q-A session, and the interactive CLI is WIP. This also serves as an example for users to build customized inference scripts.

python -m llava.eval.run_llava \
    --model-name /path/to/LLaVA-13B-v0 \
    --image-file "https://llava-vl.github.io/static/images/view.jpg" \
    --query "What are the things I should be cautious about when I visit here?"

Example output (varies in different runs):

When visiting this picturesque location with a serene lake and a wooden pier extending over the water, one should be cautious about various safety aspects. Some important considerations include:

Ensuring that the pier is structurally sound andstable, as old or weakened pier structures might not support the weight of visitors.

Being aware of the water depth around the pier and lake, as sudden drop-offs or strong currents may pose a risk to swimmers, boaters, or those who venture too close to the edge.

Staying vigilant about the presence of wildlife in the area, such as slippery, stealthy fish or other animals that might cause harm or inconvenience.

Maintaining a safe distance from the water's edge, particularly for children, elderly individuals, or those who are not strong swimmers.

Following any posted signs or guidelines related to safety and the use of the pier and surrounding areas.

By considering these safety precautions, visitors can enjoy the natural beauty of the location while minimizing risks and ensuring a safe and pleasant experience.

Evaluation

GPT-assisted Evaluation

Our GPT-assisted evaluation pipeline for multimodal modeling is provided for a comprehensive understanding of the capabilities of vision-language models. Please see our paper for more details.

Generate LLaVA responses

python model_vqa.py \
    --model-name ./checkpoints/LLaVA-13B-v0 \
    --question-file \
    playground/data/coco2014_val_qa_eval/qa90_questions.jsonl \
    --image-folder \
    /path/to/coco2014_val \
    --answers-file \
    /path/to/answer-file.jsonl

Evaluate the generated responses. In our case, answer-file-1.jsonl is the response generated by text-only GPT-4 (0314), with the context captions/boxes provided.

OPENAI_API_KEY="sk-***********************************" python eval_gpt_review_visual.py \
    --question playground/data/coco2014_val_qa_eval/qa90_questions.jsonl \
    --context table/caps_boxes_coco2014_val_80.jsonl \
    --answer-list \
    /path/to/answer-file-1.jsonl \
    /path/to/answer-file-2.jsonl \
    --rule table/rule.json \
    --output /path/to/review.json

Summarize the evaluation results

python summarize_gpt_review.py

ScienceQA

Prepare Data

Please see ScienceQA repo for setting up the dataset.
Generate ScienceQA dataset for LLaVA conversation-style format.

python scripts/convert_sqa_to_llava \
    convert_to_llava \
    --base-dir /path/to/ScienceQA/data/scienceqa \
    --split {train,val,minival,test,minitest}

Evaluation

Download our pretrained LLaVA-13B (delta) weights for ScienceQA dataset here. Convert the delta weights to actual weights.

python -m llava.model.apply_delta \
    --base /path/to/llama-13b \
    --target /path/to/LLaVA-13b-v0-science_qa \
    --delta liuhaotian/LLaVA-13b-delta-v0-science_qa

[Option 1] Multiple-GPU inference You may evaluate this with multiple GPUs, and concatenate the generated jsonl files. Please refer to our script for batch evaluation and results gathering.
[Option 2] Single-GPU inference

(a) Generate LLaVA responses on ScienceQA dataset

python -m llava.eval.model_vqa_science \
    --model-name /path/to/LLaVA-13b-v0-science_qa \
    --question-file /path/to/ScienceQA/data/scienceqa/llava_test.json \
    --image-folder /path/to/ScienceQA/data/scienceqa/images/test \
    --answers-file vqa/results/ScienceQA/test_llava-13b.jsonl \
    --answer-prompter
    --conv-mode simple

(b) Evaluate the generated responses

python eval_science_qa.py \
    --base-dir /path/to/ScienceQA/data/scienceqa \
    --result-file vqa/results/ScienceQA/test_llava-13b.jsonl \
    --output-file vqa/results/ScienceQA/test_llava-13b_output.json \
    --output-result vqa/results/ScienceQA/test_llava-13b_result.json \

For reference, we attach our prediction file test_llava-13b_result.json here for comparison when reproducing our results, as well as for further analysis in detail.

Fine-tuning

Data

The current version of LLaVA is fine-tuned from a Vicuna-13B model. We use approximately 600K filtered CC3M in feature alignment pretraining and 150K GPT-generated multimodal instruction-following data in finetuning. For detailed description of the data generation pipeline, please refer see our paper.

We are working on a more capable model that is pretrained with the data at a larger scale. Stay tuned!

We release all three types of multimodal instruction-following data. The use of these data is subject to OpenAI TOS.

Code and Hyperparameters

We fine-tune the model using the code from FastChat. We use a similar set of hyperparameters as Vicuna in finetuning. Both hyperparameters used in pretraining and finetuning are provided below.

Pretraining

Hyperparameter	Global Batch Size	Learning rate	Epochs	Max length	Weight decay
LLaVA-13B	128	2e-3	1	2048	0

Finetuning

Hyperparameter	Global Batch Size	Learning rate	Epochs	Max length	Weight decay
LLaVA-13B	32	2e-5	3	2048	0

Fine-tuning with Local GPUs

LLaVA is trained on 8 A100 GPUs with 80GB memory with the following code. To train on fewer GPUs, you can reduce the per_device_train_batch_size and increase the gradient_accumulation_steps accordingly to keep the global batch size the same.

Pretraining

Pretrain: LLaVA-13B, 8x A100 (80G). Time: ~4 hours.

torchrun --nnodes=1 --nproc_per_node=8 --master_port=25001 \
    llava/train/train_mem.py \
    --model_name_or_path ./checkpoints/llama-vicuna-13b \
    --data_path /path/to/cc3m_595k.json \
    --image_folder /path/to/cc3m_595k \
    --vision_tower openai/clip-vit-large-patch14 \
    --tune_mm_mlp_adapter True \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end \
    --bf16 True \
    --output_dir ./checkpoints/llava-13b-pretrain \
    --num_train_epochs 1 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 2400 \
    --save_total_limit 1 \
    --learning_rate 2e-3 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --lazy_preprocess True \
    --report_to wandb

You may run this with a single A100 GPU with the following code. Please note that the per_device_train_batch_size * gradient_accumulation_steps should be equal to 128 to keep the global batch size the same.

Pretrain: LLaVA-13B, 1x A100 (80G). Time: ~33 hours.

python llava/train/train_mem.py \
    --model_name_or_path ./checkpoints/llama-vicuna-13b \
    --data_path /path/to/cc3m_595k.json \
    --image_folder /path/to/cc3m_595k \
    --vision_tower openai/clip-vit-large-patch14 \
    --tune_mm_mlp_adapter True \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end \
    --bf16 True \
    --output_dir ./checkpoints/llava-13b-pretrain \
    --num_train_epochs 1 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 8 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 2400 \
    --save_total_limit 1 \
    --learning_rate 2e-3 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --lazy_preprocess True \
    --report_to wandb

Pretrain: LLaVA-7B, 1x A100 (80G/40G). Time: ~19 hours.

python llava/train/train_mem.py \
    --model_name_or_path ./checkpoints/llama-vicuna-7b \
    --data_path /path/to/cc3m_595k.json \
    --image_folder /path/to/cc3m_595k \
    --vision_tower openai/clip-vit-large-patch14 \
    --tune_mm_mlp_adapter True \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end \
    --bf16 True \
    --output_dir ./checkpoints/llava-7b-pretrain \
    --num_train_epochs 1 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 8 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 2400 \
    --save_total_limit 1 \
    --learning_rate 2e-3 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --lazy_preprocess True \
    --report_to wandb

Experimental: use FSDP to save memory in pretraining

Learn more

Currently, PyTorch and Huggingface does not yet have stable/native support for FSDP on parameter efficient tuning (part of the parameters are frozen). However, the feature is being developed in PyTorch nightly and shall be shipped in the next release. We provide an experimental script to enable FSDP in pretraining. To use it, please create a new enviroment (to be safe), install PyTorch nightly (MUST), and LLaVA package following the instructions below.

Prepare environment

conda create -n llava_beta python=3.10 -y
conda activate llava_beta
pip install --upgrade pip
pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu117
pip install -e .
pip install einops ninja
pip install flash-attn

Run pretraining with FSDP (experimental)

torchrun --nnodes=1 --nproc_per_node=8 --master_port=25001 \
    llava/train/train_mem.py \
    --model_name_or_path ./checkpoints/llama-vicuna-13b \
    --data_path /path/to/cc3m_595k.json \
    --image_folder /path/to/cc3m_595k \
    --vision_tower openai/clip-vit-large-patch14 \
    --tune_mm_mlp_adapter True \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end \
    --bf16 True \
    --output_dir ./checkpoints/llava-13b-pretrain_fsdp \
    --num_train_epochs 1 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 2400 \
    --save_total_limit 1 \
    --learning_rate 2e-3 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --lazy_preprocess True \
    --report_to wandb

Extract projector features

python scripts/extract_mm_projector.py \
  --model_name_or_path ./checkpoints/llava-13b-pretrain \
  --output ./checkpoints/mm_projector/llava-13b-pretrain.bin

Finetuning

torchrun --nnodes=1 --nproc_per_node=8 --master_port=25001 \
    llava/train/train_mem.py \
    --model_name_or_path /path/to/llama-vicuna-13b \
    --data_path /path/to/llava_instruct_150k.json \
    --image_folder /Data/haotian/coco/train2014 \
    --vision_tower openai/clip-vit-large-patch14 \
    --pretrain_mm_mlp_adapter ./checkpoints/mm_projector/llava-13b-pretrain.bin \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end True \
    --bf16 True \
    --output_dir ./checkpoints \
    --num_train_epochs 3 \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 5000 \
    --save_total_limit 3 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --lazy_preprocess True \
    --report_to wandb

Train LLaVA Lightning

LLaVA-Lightning can be trained on 8x A100 GPUs in just 3 hours, including both pretraining and finetuning. When using spot instances, it costs just ~$40. We are working on SkyPilot tutorial to make spot instance training even easier, stay tuned!

Please make sure to: (1) install or upgrade to the latest code base, and (2) pass the correct model version identifier v0/v1 to ensure the correct conversation template is loaded.

bash ./scripts/train_lightning.sh {v0,v1}

Hyperparameters

Pretraining

Hyperparameter	Global Batch Size	Learning rate	Epochs	Max length	Weight decay
LLaVA-Lightning-7B	128	2e-3	1	2048	0

Finetuning

Hyperparameter	Global Batch Size	Learning rate	Epochs	Max length	Weight decay
LLaVA-Lightning-7B	128	2e-5	1	2048	0

LLaVA-MPT-7b

Thanks to LLaVA-Lightning, we are able to train a checkpoint based on MPT-7b-Chat on 8x A100 GPUs in just 3 hours, including both pretraining and finetuning.

NOTE: This is a research preview of the LLaVA-Lightning based on MPT-7B-chat checkpoint. The usage of the model should comply with MPT-7B-chat license and agreements.

NOTE: Unlike other LLaVA models, this model should be used directly without delta weights conversion!

NOTE: You need to upgrade to our latest code base to use LLaVA-MPT-7b!

Usage

You do not need to download our checkpoint, it will directly load from our Hugging Face model: liuhaotian/LLaVA-Lightning-MPT-7B-preview.

python -m llava.serve.controller --host 0.0.0.0 --port 10000
python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path liuhaotian/LLaVA-Lightning-MPT-7B-preview
python -m llava.serve.gradio_web_server --controller http://localhost:10000

Training

We use the same set of training dataset, and the hyperparameters as other Lightning checkpoints.

bash ./scripts/train_lightning_mpt.sh

Fine-tuning on ScienceQA

NOTE: Due to that ScienceQA experiments were done earlier, the current checkpoints are trained without <im_start> and <im_end> tokens. Checkpoints with these tokens will be updated later. Here we provide our training scripts for the current checkpoints.

1. Pretraining

torchrun --nnodes=1 --nproc_per_node=8 --master_port=25001 \
    llava/train/train_mem.py \
    --model_name_or_path ./checkpoints/llama-vicuna-13b \
    --data_path /path/to/cc3m_595k.json \
    --image_folder /path/to/cc3m_595k \
    --vision_tower openai/clip-vit-large-patch14 \
    --tune_mm_mlp_adapter True \
    --mm_vision_select_layer -2 \
    --bf16 True \
    --output_dir ./checkpoints/llava-13b-pretrain-no_im_start_end_token \
    --num_train_epochs 1 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 2400 \
    --save_total_limit 1 \
    --learning_rate 2e-3 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --lazy_preprocess True \
    --report_to wandb

2. Extract projector features

python scripts/extract_mm_projector.py \
  --model_name_or_path ./checkpoints/llava-13b-pretrain-no_im_start_end_token \
  --output ./checkpoints/mm_projector/llava-13b-pretrain-no_im_start_end_token.bin

3. Finetuning

You may download our pretrained llava-13b-pretrain-no_im_start_end_token.bin here.

torchrun --nnodes=1 --nproc_per_node=8 --master_port=25001 \
    llava/train/train_mem.py \
    --model_name_or_path /path/to/llama-vicuna-13b \
    --data_path /path/to/scienceqa/llava_train_QCM-LEPA.json \
    --image_folder /path/to/scienceqa/images/train \
    --vision_tower openai/clip-vit-large-patch14 \
    --pretrain_mm_mlp_adapter ./checkpoints/mm_projector/llava-13b-pretrain-no_im_start_end_token.bin \
    --mm_vision_select_layer -2 \
    --bf16 True \
    --output_dir ./checkpoints/llava-13b-pretrain-no_im_start_end_token-finetune_scienceqa \
    --num_train_epochs 12 \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 5000 \
    --save_total_limit 3 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --lazy_preprocess True \
    --report_to wandb

Acknowledgement

Vicuna: the codebase we built upon, and our base model Vicuna-13B that has the amazing language capabilities!

If you find LLaVA useful for your your research and applications, please cite using this BibTeX:

@misc{liu2023llava,
      title={Visual Instruction Tuning}, 
      author={Liu, Haotian and Li, Chunyuan and Wu, Qingyang and Lee, Yong Jae},
      publisher={arXiv:2304.08485},
      year={2023},
}

Related Projects

Instruction Tuning with GPT-4

For future project ideas, pleae check out:

SEEM: Segment Everything Everywhere All at Once
Grounded-Segment-Anything to detect, segment, and generate anything by marrying Grounding DINO and Segment-Anything.

Name		Name	Last commit message	Last commit date
Latest commit History 125 Commits
.github/workflows		.github/workflows
images		images
llava.egg-info		llava.egg-info
llava		llava
.dockerignore		.dockerignore
Dockerfile		Dockerfile
KoLLaVA-Kovicuna-7b_inference_test.ipynb		KoLLaVA-Kovicuna-7b_inference_test.ipynb
LICENSE		LICENSE
README.md		README.md
main.py		main.py
model.py		model.py
pyproject.toml		pyproject.toml

License

movie5/movie-s-KoLLaVA

Folders and files

Latest commit

History

Repository files navigation

🏔️ REACT로 보여주는 KoLLaVA

Update Logs

Visual Chat Example

Contents

Visual Instruction Dataset

Pretraining Dataset

Install

Inference

결과 예시

Training

Pretrain

Visual instruction tuning (Finetune)

Serving

Web UI 데모 실행 방법

To-do

Team

🌋 LLaVA: Large Language and Vision Assistant

Contents

Data Download

Pretraining Dataset

GPT-4 Prompts

Install

Upgrade to v0.1

LLaVA Weights

LLaVA-13B

LLaVA-7B

LLaVA pretrained projector weights

Serving

Web UI

Launch a controller

Launch a model worker

Launch a model worker (Multiple GPUs, when GPU VRAM <= 24GB)

Launch a gradio web server.

You can open your browser and chat with a model now.

CLI Inference

Evaluation

GPT-assisted Evaluation

ScienceQA

Prepare Data

Evaluation

Fine-tuning

Data

Code and Hyperparameters

Fine-tuning with Local GPUs

Experimental: use FSDP to save memory in pretraining

Train LLaVA Lightning

Hyperparameters

LLaVA-MPT-7b

Fine-tuning on ScienceQA

Acknowledgement

Related Projects

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages