Memory leak when trying to read objects from the api at /v1/objects

[ciantfv456]

Describe the bug

We have a script that queries all of the services on icinga2 from the API.
The memory usage keeps increasing every time we access the /v1/objects route on the API.

To Reproduce

Provide a link to a live example, or an unambiguous set of steps to reproduce this bug. Include configuration, logs, etc. to reproduce, if relevant.

1. Run icinga2 2.14 on container
2. Create 40000 services
3. Query the API at /v1/objects multiple times
4. See the memory increasing

Expected behavior

The memory that was allocated to the query should be set free after sending back the request

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version): 2.14.0
• Operating System and version: container running on centos 7
• Enabled features (icinga2 feature list): icingadb api checker
• Icinga Web 2 version and modules (System - About): 2.14.1

[Jamess11]

Hey, it happend to me too. It seems like icinga2 doesn't free the memory allocated when querying the API for objects. When it queries a lot of objects periodically the memory usage keeps increasing until the vm gets oom kill

[Al2Klimov]

Hello and thank you for reporting!

Would you try this PR if I build Icinga 2 for you?

• /v1/debug/malloc_trim: call malloc_trim(3) if available #9998

Best,
A/K

[yhabteab]

Query the API at /v1/objects multiple times

Hi @ciantfv456, did you query all 40K services unfiltered? If so, did your client fully read the response before closing the connection? We have already identified a memory leak occurring when a large number of objects are queried and the client prematurely closes the stream, before fully reading the responses, which has been fixed by #10006 and will be released in the coming weeks. Nevertheless, we are also observing a situation where Icinga 2 is allocating a significant amount of memory, for example, due to excessive API usage and not returning it to the OS when it is no longer needed, but holding it back for later uses instead. In this case, using malloc_trim() as Alex suggested above forces Icinga 2 to return the memory to the OS, but it is unfeasible to require this from an end user.

You can however replace the default Icinga 2 memory allocator by jemalloc as described here, which significantly reduces memory footprint in our testing system, though these are not real world setups. It would therefore be very valuable to receive your feedback in this regard!

[ciantfv456]

Thanks, @yhabteab.
Worked like a charm😁

[yhabteab]

Thanks, @yhabteab. Worked like a charm😁

Does this mean that Icinga 2 no longer suffers from memory leaks when linked to jemalloc? If that indeed helped you to resolve your memory issue, then @Jamess11 may also want to consider trying this out.