Single pipeline for layout and image generation

[MuhammadHammadBashir]

I am looking at example.py it only generates the images with predefined coordinates as input.
Do we need to use multiple LLM (one to produce coordinates and one to generate image).
Can you provide your thoughts how I can work with both, do we need to run.

What are differences between multilayers generation, testing multi layer generation and LLM layout for planning?

[MuhammadHammadBashir]

Another thing similar to above thread,
I downloaded both base and checkout planner models for the use of llm layout,
latest tokenzier size: 129423
Resuming from checkpoint layout_planner_checkpoint, Waiting to ready
Using Meta-Llama-3-8B for the language model.

wouldn't load_in_4bit
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with model.to('cuda').
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/workspace/art-msra/layout_planner/inference_layout.py", line 158, in
model, quantizer, tokenizer = buildmodel(device=device, **params_dict)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/art-msra/layout_planner/inference_layout.py", line 60, in buildmodel
model = LayoutModel.from_pretrained(resume, config=model_args).to(device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/transformers/modeling_utils.py", line 3404, in from_pretrained
model = cls(config, *model_args, **model_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/art-msra/layout_planner/models/modeling_layout.py", line 125, in init
self.lm = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/transformers/modeling_utils.py", line 3531, in from_pretrained
) = cls._load_pretrained_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/transformers/modeling_utils.py", line 3938, in _load_pretrained_model
state_dict = load_state_dict(shard_file, is_quantized=is_quantized)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/transformers/modeling_utils.py", line 506, in load_state_dict
with safe_open(checkpoint_file, framework="pt") as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge

After running inference script finding this error

[yifanpu001]

I am looking at example.py it only generates the images with predefined coordinates as input. Do we need to use multiple LLM (one to produce coordinates and one to generate image). Can you provide your thoughts how I can work with both, do we need to run.

What are differences between multilayers generation, testing multi layer generation and LLM layout for planning?

Thank you for your interest in our work.

This repository contains two models: (1) the first is a Layout LLM, which receive image caption and produce the predicted layout (coordinates); (2) the second is a multi-layer image generation model, which takes image caption and layout as input, and generate multi-layer image.

If you want a fully automated multi-layer image generation pipeline (where the user only provides an image caption), you should run both models. However, if you can provide both the image caption and layout yourself, you only need to use the multi-layer image generation model.

[MuhammadHammadBashir]

Hi @yifanpu001 thanks for quick and detailed reply. you are explaining nicely.
can you point out the solution to error while trying to run the layout planner.

[yifanpu001]

Hi @yifanpu001 thanks for quick and detailed reply. you are explaining nicely. can you point out the solution to error while trying to run the layout planner.

Appreciate it! Regarding the error in layout planner, we are currently working on reproducing the issue. Please allow us some time to investigate further.

[MuhammadHammadBashir]

again thanks for feedback, looking forward to your response

[MuhammadHammadBashir]

Hi @yifanpu001 is there any update on this part?