Possible memory leak

[janusn]

Check the bug
Before filling this bug report, please search if a similar issue already exists.
In this case, just add a comment on this existing issue.

Describe the bug
A clear and concise description of what the bug is.

The memory usage reported by docker stats keeps increasing on 3 separated instances running on 3 different machines.

To Reproduce
Steps to reproduce the behavior:

1. Run the following command as root:
   # docker compose up -d
2. add a line to the crontab of the root:

* * * * * /usr/bin/docker stats --no-stream --format '{{.MemUsage}}' glances | cut -d '/' -f 1 >>/var/log/docker_stats/glances.log

3. Monitor the file /var/log/docker_stats/glances.log

Expected behavior
A clear and concise description of what you expected to happen.

I expect the memory logged in the glances.log to be stable over time.

Screenshots
If applicable, add screenshots to help explain your problem.

Environement (please complete the following information)

• Operating System (lsb_release -a or OS name/version): Debian GNU/Linux 12 (bookworm)
• Glances & psutil versions:

Glances API version:	4
PsUtil version:		6.0.0
Log file:		/tmp/glances-root.log

• How do you install Glances (Pypi package, script, package manager, source): docker compose
• Glances test:

===============================================================================
Glances 4.1.2 (/app/glances/__init__.py)
Python 3.11.9 (/venv/bin/python3)
PsUtil 6.0.0 (/venv/lib/python3.11/site-packages/psutil/__init__.py)
===============================================================================
alert         [OK]    0.00003s 
[]
amps          [OK]    0.00023s key=name 
[{'count': 0,
  'countmax': None,
  'countmin': 1.0,
  'key': 'name',
  'name': 'Dropbox',
  'refresh': 3.0,
  'regex': True,
  'result': None,
  'timer': 3.0309574604034424}, ...]
cloud         [NA]
connections   [NA]
containers    [OK]    0.29408s key=name 
[{'command': '/init',
  'cpu_percent': 100.52314814814815,
  'created': '2024-10-12T12:44:28.065151703Z',
  'engine': 'docker',
  'id': '9578831f53f31e32fb31428c91d871545fee2d0111ac184aebfe27c9a1e1ee29',
  'image': ('linuxserver/bazarr:latest',),
  'io_rx': None,
  'io_wx': None,
  'key': 'name',
  'memory_percent': None,
  'memory_usage': 323547136,
  'name': 'bazarr',
  'network_rx': None,
  'network_tx': None,
  'status': 'running',
  'uptime': 'yesterday'}, ...]
core          [OK]    0.00713s 
{'log': 48, 'phys': 24}
cpu           [OK]    0.00180s 
{'cpucore': 48,
 'ctx_switches': 0,
 'ctx_switches_gauge': 3156258973,
 'guest': 0.0,
 'idle': 96.2,
 'interrupts': 0,
 'interrupts_gauge': 1395350189,
 'iowait': 0.0,
 'irq': 0.0,
 'nice': 0.0,
 'soft_interrupts': 0,
 'soft_interrupts_gauge': 1355389962,
 'steal': 0.0,
 'syscalls': 0,
 'syscalls_gauge': 0,
 'system': 1.9,
 'time_since_update': 2.3210175037384033,
 'total': 4.2,
 'user': 1.8}
diskio        [OK]    0.00596s key=disk_name 
[{'disk_name': 'nvme0n1',
  'key': 'disk_name',
  'read_bytes': 0,
  'read_bytes_gauge': 19937036288,
  'read_count': 0,
  'read_count_gauge': 919639,
  'time_since_update': 2.326836109161377,
  'write_bytes': 0,
  'write_bytes_gauge': 50798579712,
  'write_count': 0,
  'write_count_gauge': 4789946}, ...]
folders       [OK]    0.00005s 
[]
fs            [OK]    0.00130s key=mnt_point 
[{'device_name': 'tank/docker',
  'free': 125379863248896,
  'fs_type': 'zfs',
  'key': 'mnt_point',
  'mnt_point': '/config',
  'percent': 0.0,
  'size': 125394844778496,
  'used': 14981529600}, ...]
gpu           [OK]    0.00004s 
[]
help          [OK]    0.00001s 
None
ip            [OK]    0.00233s 
{'address': '***',
 'gateway': '***',
 'mask': '***',
 'mask_cidr': '***',
 'public_address': '***',
 'public_info_human': '***'}
irq           [NA]
load          [OK]    0.00006s 
{'cpucore': 48, 'min1': 2.109375, 'min15': 1.20703125, 'min5': 1.46728515625}
mem           [OK]    0.00028s 
{'active': 870154240,
 'available': 61882241024,
 'buffers': 15302656,
 'cached': 2608340992,
 'free': 61882241024,
 'inactive': 8887496704,
 'percent': 77.1,
 'shared': 180944896,
 'total': 270106365952,
 'used': 208224124928}
memswap       [OK]    0.00052s 
{'free': 0, 'percent': 0.0, 'sin': 0, 'sout': 0, 'time_since_update': 2.337296962738037, 'total': 0, 'used': 0}
network       [OK]    0.00664s key=interface_name 
[{'alias': None,
  'bytes_all': 200,
  'bytes_all_gauge': 6824554,
  'bytes_all_rate_per_sec': 86.0,
  'bytes_recv': 100,
  'bytes_recv_gauge': 3412277,
  'bytes_recv_rate_per_sec': 43.0,
  'bytes_sent': 100,
  'bytes_sent_gauge': 3412277,
  'bytes_sent_rate_per_sec': 43.0,
  'interface_name': 'lo',
  'key': 'interface_name',
  'speed': 0,
  'time_since_update': 2.3232762813568115}, ...]
now           [OK]    0.00006s 
{'custom': '2024-10-13 22:38:44 BST', 'iso': '2024-10-13T22:38:44+01:00'}
percpu        [OK]    0.00266s key=cpu_number 
[{'cpu_number': 0,
  'dpc': None,
  'guest': 0.0,
  'guest_nice': 0.0,
  'idle': 96.4,
  'interrupt': None,
  'iowait': 1.1,
  'irq': 0.0,
  'key': 'cpu_number',
  'nice': 0.7,
  'softirq': 0.0,
  'steal': 0.0,
  'system': 0.4,
  'total': 3.6,
  'user': 1.4}, ...]
ports         [OK]    0.00001s 
[]
processcount  [OK]    0.12034s 
{'pid_max': 0, 'running': 1, 'sleeping': 288, 'thread': 1826, 'total': 289}
processlist   [OK]    0.00108s 
[]
psutilversion [OK]    0.00017s 
'6.0.0'
quicklook     [OK]    0.00151s 
{'cpu': 4.2,
 'cpu_hz': 3000000000.0,
 'cpu_hz_current': 859181958.3333333,
 'cpu_log_core': 48,
 'cpu_name': 'Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz',
 'cpu_phys_core': 24,
 'load': 2.5,
 'mem': 77.1,
 'percpu': [{...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...},
            {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...},
            {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...},
            {...}, {...}, {...}],
 'swap': 0.0}
raid          [NA]
sensors       [OK]    0.00008s key=label 
[{'critical': 79,
  'key': 'label',
  'label': 'Composite',
  'type': <SensorType.CPU_TEMP: 'temperature_core'>,
  'unit': 'C',
  'value': 38,
  'warning': 74}, ...]
smart         [NA]
system        [OK]    0.00002s 
{'hostname': 'rocker',
 'hr_name': 'Alpine Linux 3.19.2 64bit / Linux 6.6.32-production+truenas',
 'linux_distro': 'Alpine Linux 3.19.2',
 'os_name': 'Linux',
 'os_version': '6.6.32-production+truenas',
 'platform': '64bit'}
uptime        [OK]    0.00044s 
{'seconds': 284581}
version       [OK]    0.00013s 
'4.1.2'
wifi          [OK]    0.00008s 
[]
===============================================================================
Total time to update all stats: 0.45255s
===============================================================================

Additional context
Add any other context about the problem here.

Here is a sample of the content of the file glances.log after 2 days of running:

112.4MiB 
113.4MiB 
113.4MiB 
113.4MiB 
113.4MiB 
113.1MiB 
113MiB 
113.5MiB 
113.5MiB 
113.5MiB 
113.5MiB 
113.5MiB 
113.4MiB 
* snip … snip *
193.8MiB 
194MiB 
194MiB 
194MiB 
194.1MiB 
194.2MiB 
194.3MiB 
194.3MiB 
194.4MiB 
194.5MiB 
194.7MiB 
194.6MiB 
194.6MiB 
194.6MiB 
194.6MiB 
194.8MiB 

the content of the compose.yaml:

---
#version: "3"
services:
  glances:
    image: nicolargo/glances:latest-full
    container_name: glances
    privileged: true
    deploy:
      resources:
        limits:
          memory: 512M
    pid: host
    network_mode: host
    environment:
      - "TZ=Europe/London"
      - "GLANCES_OPT=-C /config/glances.conf -w --password -u ha"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config:/config:ro
    secrets:
      - source: ha_password
        target: /root/.config/glances/ha.pwd
    restart: unless-stopped
secrets:
  ha_password:
    file: ./secrets/ha_password.txt

modification of glances.conf:

[gpu]
disable=True
[hddtemp]
disable=True
[graph]
generate_every=0
[diskio]
hide=loop.*,/dev/loop.*,sd*
[fs]
hide=/boot.*,/snap.*,/etc/*,_config/*

You can also pastebin:

• the Glances configuration file (https://glances.readthedocs.io/en/latest/config.html#location)
• the Glances log file (https://glances.readthedocs.io/en/latest/config.html#logging)

[nicolargo]

Hi @janusn

in the Glances configuration file you have the following key:

[global]
# History size (maximum number of values)
# Default is 1200 values (~1h with the default refresh rate)
history_size=1200

So it's a "normal" behavior to have your memory increase for the first hour.

After that it should be stable.

[janusn]

I have been monitoring the memory consumption since your reply. There were a few set backs such as container recreated by watchtower. Nonetheless, here is what I have found.

The memory usage on 2 of 3 instances of the glances keep growing over 24 hours period but the other one does not.
Here is a summary of memory usage of one of the growing instances:

73.51MiB (runtime 00:00 hours)
104.9MiB (runtime 01:00 hour)
105.3MiB (runtime 02:00 hours)
107.7MiB (runtime 03:00 hours)
108.3MiB (runtime 04:00 hours)
109.8MiB (runtime 05:00 hours)
110MiB (runtime 06:00 hours)
112.3MiB (runtime 07:00 hours)
113.9MiB (runtime 08:00 hours)
114.7MiB (runtime 09:00 hours)
115MiB (runtime 10:00 hours)
117.1MiB (runtime 11:00 hours)
116.1MiB (runtime 12:00 hours)
117MiB (runtime 13:00 hours)
117.3MiB (runtime 14:00 hours)
119.1MiB (runtime 15:00 hours)
120.7MiB (runtime 16:00 hours)
121.5MiB (runtime 17:00 hours)
121.1MiB (runtime 18:00 hours)
121.7MiB (runtime 19:00 hours)
122.2MiB (runtime 20:00 hours)
122.6MiB (runtime 21:00 hours)
124.8MiB (runtime 22:00 hours)
124.5MiB (runtime 23:00 hours)
126MiB  (runtime 24:00 hours)
126.2MiB (runtime 25:00 hours)
125.4MiB (runtime 26:00 hours)
126.3MiB (runtime 27:00 hours)
127.4MiB (runtime 28:00 hours)
127.6MiB (runtime 29:00 hours)
129.7MiB (runtime 30:00 hours)
130.4MiB (runtime 31:00 hours)
130.4MiB (runtime 32:00 hours)
132.2MiB  (runtime 33:00 hours)

[nicolargo]

On my configuration (running Glances in local outside a Docker container) and with a history_size=3.

When Glances start:

rss=64245760, vms=590016512, shared=17039360, text=3026944, lib=0, data=91455488, dirty=0, uss=51081216, pss=52712448, swap=0

One hour later:

rss=65421312, vms=590155776, shared=17039360, text=3026944, lib=0, data=91594752, dirty=0, uss=51511296, pss=53140480, swap=0

So memory increase around 1.4 MB (regarding 31 MB on your test).

I will make a long term test next week.

[nicolargo]

Also reproduced on my side (in console mode with default history_size=1200) during 30 hours but with a more limited leak:

Test with history_size=0 (in red):

[nicolargo]

Note for myself: https://pythonhosted.org/Pympler/muppy.html

[nicolargo]

Ok found it thanks to mem_top:

2024-10-31 08:40:16,583 -- INFO -- 
refs:
1519	<class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386	<class 'dict'> {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776: 
870	<class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
870	<class 'dict'> {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
870	<class 'dict'> {3119283: {'cpu_percent': {'decoration': 'DEFAULT', 'optional': False, 'additional': False, 'splitta
756	<class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
523	<class 'collections.defaultdict'> defaultdict(<class 'int'>, {('sda', 0): 0, ('sda', 1): 0, ('sda', 2): 0, ('sda', 3): 0, ('sda', 4): 
450	<class 'list'> [(0, 0, 0), (1, 31, -1), (127, 159, -1), (173, 173, 0), (768, 879, 0), (1155, 1161, 0), (1425, 1469,
435	<class 'list'> [{'cpu_percent': 12.2, 'num_threads': 1, 'name': 'python', 'memory_percent': 0.19223070159317707, 'g
423	<class 'dict'> {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions

bytes:
73816	 {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960	 {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776: 
36960	 {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
18520	 {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520	 {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520	 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520	 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520	 {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python.  Normal
18520	 {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa
18520	 {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi

types:
11694	 <class 'function'>
9483	 <class 'dict'>
7183	 <class 'tuple'>
2653	 <class 'weakref'>
2237	 <class 'list'>
2050	 <class 'cell'>
2003	 <class 'getset_descriptor'>
1574	 <class 'type'>
1394	 <class 'wrapper_descriptor'>
1339	 <class 'builtin_function_or_method'>

===

2024-10-31 09:26:12,157 -- INFO -- 
refs:
1690	<class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
1519	<class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386	<class 'dict'> {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776: 
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 551316), 2.0), (datetime.datetime(2024, 10, 31, 8, 40, 
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 551311), 7.1), (datetime.datetime(2024, 10, 31, 8, 40, 
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 552876), 64.8), (datetime.datetime(2024, 10, 31, 8, 40,
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 552878), 84.9), (datetime.datetime(2024, 10, 31, 8, 40,
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 553628), 0.74), (datetime.datetime(2024, 10, 31, 8, 40,
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 553631), 0.64), (datetime.datetime(2024, 10, 31, 8, 40,
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 556058), 1.3), (datetime.datetime(2024, 10, 31, 8, 40, 

bytes:
73816	 {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960	 {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776: 
36960	 {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
36960	 {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
36960	 {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
18520	 {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520	 {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520	 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520	 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520	 {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python.  Normal

types:
11694	 <class 'function'>
9905	 <class 'dict'>
8377	 <class 'tuple'>
3985	 <class 'list'>
2651	 <class 'weakref'>
2050	 <class 'cell'>
2003	 <class 'getset_descriptor'>
1574	 <class 'type'>
1394	 <class 'wrapper_descriptor'>
1338	 <class 'builtin_function_or_method'>

===

AFTER a F5 (clear processlist cache)

2024-10-31 09:31:11,215 -- INFO -- 
refs:
1519	<class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386	<class 'dict'> {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776: 
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 456881), 0.5), (datetime.datetime(2024, 10, 31, 8, 45, 
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 456876), 2.4), (datetime.datetime(2024, 10, 31, 8, 45, 
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 458499), 64.8), (datetime.datetime(2024, 10, 31, 8, 45,
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 458501), 84.9), (datetime.datetime(2024, 10, 31, 8, 45,
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 459220), 0.52), (datetime.datetime(2024, 10, 31, 8, 45,
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 459222), 0.56), (datetime.datetime(2024, 10, 31, 8, 45,
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 461633), 1.3), (datetime.datetime(2024, 10, 31, 8, 45, 
1200	<class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 461635), 0.4), (datetime.datetime(2024, 10, 31, 8, 45, 

bytes:
73816	 {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960	 {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776: 
36960	 {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
18520	 {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520	 {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520	 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520	 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520	 {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python.  Normal
18520	 {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa
18520	 {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi

types:
11694	 <class 'function'>
9433	 <class 'dict'>
8377	 <class 'tuple'>
3552	 <class 'list'>
2651	 <class 'weakref'>
2050	 <class 'cell'>
2003	 <class 'getset_descriptor'>
1574	 <class 'type'>
1394	 <class 'wrapper_descriptor'>
1338	 <class 'builtin_function_or_method'>

[nicolargo]

The issue is confirmed and is in the processes.py file / class GlancesProcesses / dict self.processlist_cache.

The size of this dict can be very large because the key is the PID, on Linux system the default value for this file, 32768, results in the same range of PIDs as on earlier kernels. On 32-bit platforms, 32768 is the maximum value for pid_max. On 64-bit systems, pid_max can be set to any value up to 2^22 (PID_MAX_LIMIT, approximately 4 million)...

So we need to have a peace of code after the main loop of the update method to clean key/value when PID is no more existing:

• during the loop: build the list of current PIDs or use the processlist variable (list of dict)
• after the loop, remove from the self.processlist_cache key/value if key is not in the list of current PIDs

[nicolargo]

Adding this code after the loop:

        # Remove non running process from the cache (avoid issue #2976)
        pids_running = [p['pid'] for p in processlist]
        pids_cached = [p for p in self.processlist_cache.keys()]
        for pid in pids_cached:
            if pid not in pids_running:                                                                               
                self.processlist_cache.pop(pid, None)

seems to solve the leak:

Running Glances with (history_size=0):

$ while true; do ./venv/bin/python -c 'import os; import time; import psutil; g = psutil.Process(3223060); print("{} {}".format(time.strftime("%Y-%m-%d %H:%M:%S"), g.memory_full_info()));'; sleep 300; done
2024-10-31 16:10:03 pfullmem(rss=70799360, vms=1403809792, shared=15728640, text=4096, lib=0, data=152330240, dirty=0, uss=60723200, pss=63950848, swap=0)
2024-10-31 16:15:03 pfullmem(rss=70557696, vms=1403809792, shared=15728640, text=4096, lib=0, data=152330240, dirty=0, uss=60678144, pss=63902720, swap=0)
2024-10-31 16:20:03 pfullmem(rss=72482816, vms=1405534208, shared=15728640, text=4096, lib=0, data=154054656, dirty=0, uss=62652416, pss=65876992, swap=0)
2024-10-31 16:25:03 pfullmem(rss=71524352, vms=1404329984, shared=15728640, text=4096, lib=0, data=152850432, dirty=0, uss=61501440, pss=64351232, swap=0)
2024-10-31 16:30:03 pfullmem(rss=71847936, vms=1404592128, shared=15728640, text=4096, lib=0, data=153120768, dirty=0, uss=61755392, pss=64605184, swap=0)

Also confirmed by mem_top:

2024-10-31 16:07:57,006 -- INFO -- 
refs:
1519	<class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386	<class 'dict'> {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544: 
890	<class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
890	<class 'dict'> {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
890	<class 'dict'> {16089: {'num_threads': {'decoration': 'DEFAULT', 'optional': False, 'additional': False, 'splittabl
756	<class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
523	<class 'collections.defaultdict'> defaultdict(<class 'int'>, {('sda', 0): 0, ('sda', 1): 0, ('sda', 2): 0, ('sda', 3): 0, ('sda', 4): 
450	<class 'list'> [(0, 0, 0), (1, 31, -1), (127, 159, -1), (173, 173, 0), (768, 879, 0), (1155, 1161, 0), (1425, 1469,
445	<class 'list'> [{'num_threads': 26, 'pid': 16089, 'memory_percent': 2.455672030884629, 'cpu_percent': 19.7, 'gids':
423	<class 'dict'> {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions

bytes:
73816	 {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960	 {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544: 
36960	 {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
18520	 {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520	 {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520	 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520	 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520	 {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python.  Normal
18520	 {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa
18520	 {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi

types:
11701	 <class 'function'>
10050	 <class 'dict'>
7218	 <class 'tuple'>
2693	 <class 'weakref'>
2584	 <class 'list'>
2090	 <class 'cell'>
2003	 <class 'getset_descriptor'>
1574	 <class 'type'>
1514	 <class 'builtin_function_or_method'>
1394	 <class 'wrapper_descriptor'>

===

2024-10-31 16:34:09,533 -- INFO -- 
refs:
1519	<class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386	<class 'dict'> {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544: 
926	<class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
926	<class 'dict'> {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
926	<class 'dict'> {3223060: {'num_threads': {'decoration': 'DEFAULT', 'optional': False, 'additional': False, 'splitta
756	<class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
523	<class 'collections.defaultdict'> defaultdict(<class 'int'>, {('sda', 0): 0, ('sda', 1): 0, ('sda', 2): 0, ('sda', 3): 0, ('sda', 4): 
463	<class 'list'> [{'num_threads': 9, 'pid': 3223060, 'memory_percent': 0.21623942888930556, 'cpu_percent': 14.1, 'gid
450	<class 'list'> [(0, 0, 0), (1, 31, -1), (127, 159, -1), (173, 173, 0), (768, 879, 0), (1155, 1161, 0), (1425, 1469,
423	<class 'dict'> {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions

bytes:
73816	 {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960	 {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544: 
36960	 {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
36960	 {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
18520	 {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520	 {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520	 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520	 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520	 {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python.  Normal
18520	 {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa

types:
11701	 <class 'function'>
10147	 <class 'dict'>
7218	 <class 'tuple'>
2698	 <class 'list'>
2693	 <class 'weakref'>
2090	 <class 'cell'>
2003	 <class 'getset_descriptor'>
1574	 <class 'type'>
1514	 <class 'builtin_function_or_method'>
1394	 <class 'wrapper_descriptor'>

2024-10-31 16:34:11,742 -- INFO -- 
refs:
1519	<class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386	<class 'dict'> {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544: 
926	<class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
926	<class 'dict'> {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
926	<class 'dict'> {2175857: {'num_threads': {'decoration': 'DEFAULT', 'optional': False, 'additional': False, 'splitta
756	<class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
523	<class 'collections.defaultdict'> defaultdict(<class 'int'>, {('sda', 0): 0, ('sda', 1): 0, ('sda', 2): 0, ('sda', 3): 0, ('sda', 4): 
463	<class 'list'> [{'num_threads': 22, 'pid': 2175857, 'memory_percent': 10.093652599903841, 'cpu_percent': 151.6, 'gi
450	<class 'list'> [(0, 0, 0), (1, 31, -1), (127, 159, -1), (173, 173, 0), (768, 879, 0), (1155, 1161, 0), (1425, 1469,
423	<class 'dict'> {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions

bytes:
73816	 {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960	 {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544: 
36960	 {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': 
36960	 {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
18520	 {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520	 {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520	 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520	 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520	 {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python.  Normal
18520	 {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa

types:
11701	 <class 'function'>
10148	 <class 'dict'>
7218	 <class 'tuple'>
2699	 <class 'list'>
2693	 <class 'weakref'>
2090	 <class 'cell'>
2003	 <class 'getset_descriptor'>
1574	 <class 'type'>
1514	 <class 'builtin_function_or_method'>
1394	 <class 'wrapper_descriptor'>

had to make some extra-test before pushing but look ok.

[nicolargo]

Ok pushed on the develop branch.

After some test, il will release a version 4.2.1.

[nicolargo]

@janusn Glances 4.2.1 is available. You can upgrade your system.

[janusn]

@nicolargo

---

Update:

I checked again and I realized that the image tagged 4.2.1-full is different from the image I was running, tagged latest-full. Let me run the 4.2.1-full a couple day. I will report my result afterwards

$ sudo docker image ls
REPOSITORY                        TAG           IMAGE ID       CREATED         SIZE
nicolargo/glances                 4.2.1-full    a03e9d74b587   3 days ago      197MB
nicolargo/glances                 latest-full   d9f6a9532303   3 days ago      197MB

---

Original comment:

Thanks for the quick fix.

On one of the 3 containers, the memory usage is still growing albeit it is much slower than before.

11/2
05:17:03 65.92MiB (container restarted by watchtower)
06:00:03 70.41MiB
07:00:03 79.04MiB
08:00:03 96.26MiB
09:00:03 98.54MiB
10:00:03 97.65MiB
11:00:03 97.6MiB
12:00:04 98.7MiB
13:00:03 98.39MiB
14:00:03 99.37MiB
15:00:03 99.45MiB
16:00:03 99.29MiB
17:00:04 98.3MiB
18:00:03 99.8MiB
19:00:03 99.16MiB
20:00:03 100.5MiB
21:00:03 99.81MiB
22:00:03 99.62MiB
23:00:03 99.59MiB
11/3
00:00:04 99.93MiB
01:00:04 99.63MiB
02:00:03 99.88MiB
03:00:04 99.86MiB
04:00:03 100MiB
05:00:03 100.1MiB
06:00:03 101.7MiB
07:00:03 100.5MiB
08:00:03 100.7MiB
09:00:04 100.7MiB
10:00:03 101.4MiB
11:00:03 101.4MiB
12:00:04 101.4MiB
13:00:04 101.4MiB
14:00:04 101.4MiB
15:00:03 104.2MiB
16:00:03 104.3MiB
17:00:04 104.5MiB
18:00:04 104.4MiB
19:00:03 103.5MiB
20:00:02 104.7MiB
21:00:03 104.4MiB
22:00:03 104.4MiB
23:00:03 104.6MiB
11/4
00:00:03 103.5MiB
01:00:04 103.6MiB
02:00:03 105.2MiB
03:00:03 105.3MiB
04:00:03 105.1MiB
05:00:03 104.2MiB
06:00:03 105.3MiB
07:00:03 104.3MiB
08:01:04 104.2MiB
09:00:03 105.5MiB
10:00:04 104.5MiB
11:00:04 104.5MiB
12:00:03 105.7MiB
13:00:04 105.4MiB
14:00:03 105.6MiB
15:00:03 105.4MiB
16:00:03 106.2MiB
17:00:03 106.1MiB
18:00:03 105.7MiB
19:00:03 106.2MiB
20:00:03 106.2MiB
21:00:03 106.4MiB
22:00:03 106.4MiB
23:00:04 107.7MiB
11/5
00:00:03 108.1MiB