Issues with High Latency (8-12s) & Artifacts on L40 (TensorRT) - Direct Connection

[bastionloui-lgtm]

Hi Team PeronaLive,

Following up on my previous tests, I have completely upgraded my setup to match your recommended architecture.

My Current Setup:

Hardware: NVIDIA L40 (Ada Lovelace architecture, similar to RTX 4090).

Software: Standard installation from your repo + TensorRT acceleration enabled.

Network: Direct HTTP connection via GPU IP + Port (I am NOT using any tunneling service like ngrok to rule out network bottlenecks).

Despite this optimized setup, I am facing two major issues:

1. Latency is still extremely high (8-12 seconds) Since I am using an L40 with TensorRT and a direct connection, the inference should be near-instant. However, the delay remains consistently around 8 to 12 seconds. Could you explain exactly how you achieve the low latency shown in your demos? Is there a specific buffer setting or streaming configuration in the code that needs to be adjusted for cloud environments?

2. Visual Artifacts (Glasses appear on movement) When I nod or turn my head, the model suddenly generates weird glasses on the character's face, even though the reference image does not have any. This happens consistently during head movements.

Could you please provide insights on why the latency is so high despite the correct hardware/TensorRT setup, and what causes these specific artifacts?

Thanks for your help.

[huai-chang]

For the first question: The demo I mentioned in my previous reply was achieved using the code in this repository exactly as is, without any modifications. The device used was a 4090D on a remote server. During online inference, the terminal prints the number of pending frames:

PersonaLive/webcam/vid2vid_trt.py

Line 143 in 32f401d

print("input_queue size = ", input_queue.qsize())

Under normal conditions, this value should initially increase and then drop to around 4. Does this value behave normally during your inference? Also, what is your GPU utilization?

For the second question: As I mentioned in the previous answer, this is likely because our training data consists mainly of frontal views. When the head pose deviates from the training distribution, the artifacts you described may appear. On the other hand, if the issue mentioned in your first question causes the FPS to be too low, it might also contribute to this problem.

[bastionloui-lgtm]

Hi Team,

I performed the diagnostic tests as requested on an NVIDIA L40 (Ada Lovelace) with TensorRT enabled. Here are the findings.

Regarding the Artifacts (Glasses): I understand your explanation about the training data being mostly frontal. However, as you suggested, the low FPS seems to be making these artifacts much worse.

The Main Issue: GPU Saturation & Input Queue You mentioned the input_queue size should stay around 4. In my logs, the queue starts low but quickly spikes to 81+ frames.

Plaintext

input_queue size = 7
...
input_queue size = 81 <-- Massive lag starts here (8-12s latency)
input_queue size = 71
I monitored nvidia-smi during these spikes. The GPU Utilization hits 100% consistently. Even with an L40 and the TensorRT engine built locally, the model seems too heavy to process the incoming stream in real-time, causing a bottleneck.

Eventually, the WebSocket crashes due to this lag: ERROR:root:WS Error: Cannot call "receive" once a disconnect message has been received.

Context:

Card: NVIDIA L40 (48GB VRAM)

Driver: 550.90.12 / CUDA 12.4

Connection: Tunneling via SSH, but the bottleneck is clearly the GPU processing speed (100% load).

Could you advise on how to optimize the TensorRT engine further or limit the input frame rate to prevent this buffer bloat?

Thanks.

[huai-chang]

Sorry for the late reply. You may try adjusting the frontend input FPS according to your GPU's processing speed. For example, I set the frontend FPS to 15 for the RTX 4090, and 25 for the H100. In my tests, adjusting the FPS effectively resolved the frame accumulation issue. If the problem persists, I suggest measuring the time consumed at each stage to pinpoint the exact bottleneck.

Empirically, to achieve acceptable results, the GPU compute power should be at least equivalent to an RTX 4090.