Client-side memory leak?

[phikre]

Hi,

I just started using your framework on a single PC with the single_client and single_server. Carla version is 0.9.12, GStreamer 1.14.5., ROS Melodic.

I observed that the memory utilization will continuously rise until Ubuntu crashes. This happens after ~ 6 min simulation time, with multi-camera setup correspondingly faster. With only the server running, everything is fine, which is why I conclude that the leak is somewhere on the client side.

As this seems quite fundamental, is this a known issue and is there a fix?

Thanks a lot,
Philipp

[hofbi]

Hi, so far I was not aware of this issue.

Do you use server and client on the physical same machine? If yes, it is not guaranteed that the problem is on the client side. Since the pipeline and hence the buffering on the server side only starts if a client is connected it could also be an issue on the server side.

Could you provide any measurements that show which process requires the memory?

[phikre]

Maybe it's something specific to my setup.

Yes, both run on the same physical machine. I could try to run them on separate machines and check if the issue occurs as well.

Just did a quick measurement and found that the process "/catkin_ws_teleop/devel/lib/gstreaming/rtsp_server_node __name:=rtsp_server" requires the memory. So apparently the server causes the problem.

[hofbi]

I usually work with GStreamer 1.16.x. Maybe this is an issue that has been resolved.

I will try to reproduce the issue on my machine.

[phikre]

After some search and some trial-and-error, I found that it is quite uncommon to install GStreamer 1.16 on Ubuntu 18.04, as it comes with 1.14. In fact, I didn't find a simple way to upgrade. Did you build it from source?

[hofbi]

No I use 20.04 with Noetic. Melodic still uses Python 2 which is already EOL for more than 1 year

[phikre]

I upgraded to 20.04 and Noetic now. GStreamer is 1.16.2. Also, I use a virtual Python 3.7 environment to use the carla-0.9.12-py3.7-linux-x86_64.egg. $CARLA_ROOT and $PYTHONPATH are set.

Still, memory runs full after ~6 minutes. The screenshot is taken after roughly 5 minutes simulation time.

Thanks for the support so far btw!

[hofbi]

I could reproduce this problem. I convert this discussion into an issue