Inference GPU memory

[wf1024966]

Hi, I want to know, What is the minimum GPU memory required? I had inferenced the model in GPU RTX3090 24G, but it turned out of CUDA out of memory, what can I do to reduce the GPU memory ? Thanks.

[thss15fyt]

The GPU memory usage is about 38GB. It is noted that the two main components are running individually. To save memory, you could move the Llama and stable diffusion model to cpu first, and move them to gpu when using them. For applying text-to-panel, move the Llama to gpu. For applying panel-to-text, move the stable diffusion to gpu.