incorrect attribution (possibly mmap/IO or multiprocessing?)

[nrtallent]

Thank you for your work on filprofiler. I am using version 2022.1.1 on Linux (x86-x64).

I have an issue that seems to be similar to #297.

I am trying to profile a scientific program that performs a lot of I/O (HDF5) for data that is then converted to NumPy arrays.

fil shows that 4 GB of memory are used, which seems low to me, but could be true.

However, fil attributes 99% of the memory to two initial import statements (similar to #297, but 50% each), which cannot be accurate:

*  import numpy as np # first line of code
   ...
 * from pyflextrkr.advection_radar import calc_mean_advection # user code
   ... more imports

   ... all work: I/O, numpy, etc.

Probably I/O is performed by mmaping files after opening them. strace shows 2500 calls to mmap and 1078 to brk

Questions:

1. Does fil track mmap as well as sbrk?

2. Does fil have any issue with Python multiprocessing? Does it track forks and child processes? strace shows several child processes (9 SIGCHLD). Although I set the number of processes to 1, I am guessing that the multiprocessing scheduler launches a helper by default.

Thank you for any thoughts.

[itamarst]

Thanks for reporting this, I will try to investigate. To answer your questions:

1. It tracks anonymous, private mmap()s, i.e. the ones that are morally equivalent to malloc. It doesn't do sbrk, it does higher-level APIs like malloc() etc..
2. Fil does not support multiprocessing. It's on the high-priority list for the production version (https://pythonspeed.com/sciagraph/) but given limited time I'm not sure when/if it'll make it into Fil.

[itamarst]

Just to help validate my current theory, is this a machine with a large number of CPUs? And, what libraries are you using besides NumPy?

[itamarst]

I've added a numpy-specific workaround to the latest release (2022.3.0). More broadly, tracking how much of an mmap() is actually used (since by default no RAM is used until you start writing to it, allowing sparse usage) is tracking by #308.