Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get to the bottom of all the memory leaks #150

Open
pete-eiger opened this issue May 23, 2024 · 5 comments
Open

Get to the bottom of all the memory leaks #150

pete-eiger opened this issue May 23, 2024 · 5 comments
Labels
meta:triaged This issue has been triaged (has a good description, as well as labels for priority, size and type) p1 High priority size:medium Medium type:refactor Changes not visible to users

Comments

@pete-eiger
Copy link
Contributor

Thanks to #147 , we discovered that native-tls is causing a lot of memory issues in Subgraph Radio, and while it seems that it's the main culprit so far, even after removing it we get this info from heaptrack:

total runtime: 394.24s.
calls to allocation functions: 2470456 (6266/s)
temporary memory allocations: 532012 (1349/s)
peak heap memory consumption: 26.11M
peak RSS (including heaptrack overhead): 136.47M
total memory leaked: 3.57M

This is a huge improvement over the previous report (with native-tls still in the mix), but we should get to the bottom of all the leaks and suspiciously high allocations.

@pete-eiger pete-eiger self-assigned this May 23, 2024
@pete-eiger pete-eiger added size:medium Medium p1 High priority labels May 23, 2024
@pete-eiger pete-eiger added meta:triaged This issue has been triaged (has a good description, as well as labels for priority, size and type) type:bug Something isn't working type:refactor Changes not visible to users and removed type:bug Something isn't working labels May 30, 2024
@aasseman
Copy link

aasseman commented Jun 4, 2024

On my side, the update to 1.0.5 (which removed native-tls) didn't improve things significantly:
image

@pete-eiger
Copy link
Contributor Author

pete-eiger commented Jun 4, 2024

On my side, the update to 1.0.5 (which removed native-tls) didn't improve things significantly: image

thank you @aasseman , we are seeing something similar:
image

But on the bright side, it looks like memory usage is stabilizing after some time, and the Radio is effectively freeing up memory too. The usage stopped climbing up endlessly. There are still problems though, as indicated by heaptrack, we'll get to the bottom of them soon 🎯

@aasseman
Copy link

It's stable until it isn't (still on 1.0.5):
image

@pete-eiger
Copy link
Contributor Author

It's stable until it isn't (still on 1.0.5): image

darn, I'll take a look at how our instance is doing and report back

@pete-eiger
Copy link
Contributor Author

@aasseman you should definitely update to 1.0.6 though, since that removes the dependency on native-tls entirely in the SDK

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
meta:triaged This issue has been triaged (has a good description, as well as labels for priority, size and type) p1 High priority size:medium Medium type:refactor Changes not visible to users
Projects
None yet
Development

No branches or pull requests

2 participants