cuIpcGetMemHandle triggered CUDA out of memory when I use flexflow on one gpu

[Spacecat-zwh]

I used docker image "flexflow/flexflow-cuda-12.1:latest" to run flexflow on a 24GB RTX 3090,but it generated a out of memory error：

import flexflow.serve as ff 
ff.init(num_gpus=1, memory_per_gpu=11000, zero_copy_memory_per_node=11000)
[0 - 7fe1339374c0]    0.000000 {5}{gpu}: /usr/FlexFlow/deps/legion/runtime/realm/cuda/cuda_module.cc(4745):CUDA_DRIVER_FNPTR(cuIpcGetMemHandle)(&alloc.ipc_handle, alloc.dev_ptr) = 2(CUDA_ERROR_OUT_OF_MEMORY): out of memory
Aborted

Was it because I used the wrong code? How can I fix it?

[lhr-30]

maybe you can set the memory_per_gpu to a larger number like 20000, etc.

[Spacecat-zwh]

maybe you can set the memory_per_gpu to a larger number like 20000, etc.

I tried to set the memory_per_gpu to 21000, but still got the out of memory error

import flexflow.serve as ff
ff.init(num_gpus=1,memory_per_gpu=21000, zero_copy_memory_per_node=21000)
[0 - 7f474c55f4c0]    0.000000 {5}{gpu}: /usr/FlexFlow/deps/legion/runtime/realm/cuda/cuda_module.cc(4745):CUDA_DRIVER_FNPTR(cuIpcGetMemHandle)(&alloc.ipc_handle, alloc.dev_ptr) = 2(CUDA_ERROR_OUT_OF_MEMORY): out of memory
Aborted

[lhr-30]

which model did you use? It seems that when you launch it, the OOM happens

[Spacecat-zwh]

which model did you use? It seems that when you launch it, the OOM happens

I did not load any model, just initialized the flexflow backend.