fil-profiler on armv7l

[yguel]

Hi,
We are trying to run fil-profiler on armv7l 32bit architecture.
We have seen that in the file setup.py only 64bit architectures. We would like to propose changes to support armv7l architecture specifically but any kind of 32bit architecture if possible.

At first it did not compile.

The compilation error was:

running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-march=armv7-a" "-Wno-unused-result" "-Wsign-compare" "-DNDEBUG" "-g" "-fwrapv" "-O2" "-Wall" "-g" "-fstack-protector-strong" "-Wformat" "-Werror=format-security" "-g" "-flto" "-fuse-linker-plugin" "-ffat-lto-objects" "-I/usr/include/python3.7m" "-I/usr/include/python3.7m" "-ggdb" "-fno-omit-frame-pointer" "-o" "/home/pi/filprofiler/target/release/build/filpreload-5770126b567ff4d6/out/src/_filpreload.o" "-c" "src/_filpreload.c"
      cargo:warning=src/_filpreload.c: In function ‘fil_tracer’:
      cargo:warning=src/_filpreload.c:252:24: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
      cargo:warning=                        (void *)function_id + 1);
      cargo:warning=                        ^
      cargo:warning=src/_filpreload.c: In function ‘fil_mmap_impl’:
      cargo:warning=src/_filpreload.c:459:28: error: ‘SYS_mmap’ undeclared (first use in this function); did you mean ‘SYS_mmap2’?
      cargo:warning=     return (void *)syscall(SYS_mmap, addr, length, prot, flags, fd, offset);
      cargo:warning=                            ^~~~~~~~
      cargo:warning=                            SYS_mmap2
      cargo:warning=src/_filpreload.c:459:28: note: each undeclared identifier is reported only once for each function it appears in
      exit status: 1
      --- stderr
      error occurred: Command "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-march=armv7-a" "-Wno-unused-result" "-Wsign-compare" "-DNDEBUG" "-g" "-fwrapv" "-O2" "-Wall" "-g" "-fstack-protector-strong" "-Wformat" "-Werror=format-security" "-g" "-flto" "-fuse-linker-plugin" "-ffat-lto-objects" "-I/usr/include/python3.7m" "-I/usr/include/python3.7m" "-ggdb" "-fno-omit-frame-pointer" "-o" "/home/pi/filprofiler/target/release/build/filpreload-5770126b567ff4d6/out/src/_filpreload.o" "-c" "src/_filpreload.c" with args "cc" did not execute successfully (status code exit status: 1).
    cargo rustc --lib --manifest-path filpreload/Cargo.toml --features pyo3/extension-module --release --verbose -- --crate-type cdylib
    error: cargo failed with code: 101

Here is the full compilation log:
compilation_logs_of_filprofiler_on_raspberryPi_3B.txt

From that we began to try some "not educated" trials: changing the call to SYS_mmap to SYS_mmap2 and dividing the offset by 4096, it compiled but at run we got a segfault.

Then we commented the following lines fichier filpreload/build.rs
//println!("cargo:rustc-cdylib-link-arg=-Wl,--defsym=mmap=fil_mmap_impl");
//println!("cargo:rustc-cdylib-link-arg=-Wl,--defsym=mmap64=fil_mmap_impl");

However when running it allocated too much memory. We began to dive in the code, and if you could point us to the part of the code that is specific to a 64-bit architecture, we would surely go faster to understand what needs to be changed.

• sincerely

[itamarst]

32-bit support

I basically haven't thought about 32-bit at all, but off the top of my head:

1. It's quite possible there is code that assumes usize/size_t is 64-bit.
2. As you discovered, mmap is different on 32-bit. There's different system calls, and mmap call is not the same as the mmap64 call, so they would need to be handled by different codepaths.

mmap() probably isn't involved in memory leaks, so disabling it seems like a reasonable short term approach.

High memory overhead

Fil has fairly high memory overhead. I've done a bunch of work to optimize it, so it's not too bad for desktop computers, but for a machine with 1GB RAM, I can imagine it just using too much, yes.

This is separate from 32-bit vs 64-bit.

A lot of the overhead comes from use of Rust package im's HashMap. Switching to rpds's https://docs.rs/rpds/0.10.0/rpds/map/red_black_tree_map/struct.RedBlackTreeMap.html would probably reduce memory usage a lot, at the cost of slowing things down. im has alternative map with less memory usage, but it's buggy and caused crashes...

That will help, but still, overhead scales with number of allocations.

So another approach would be to use sampling, i.e. just track 1/100th of allocations. For a one-off debugging session, you can probably just hack this in, and given a leak is likely the same allocation repeated, this would probably do the trick.

Alternatives

Here are some things you might try:

1. Scalene: https://github.com/plasma-umass/scalene (might not work on 32-bit either...)
2. Do the memory profiling on a computer with more memory, as opposed to on-device.
3. https://docs.python.org/3.9/library/tracemalloc.html but only if you know all code is using Python's memory allocation APIs, it won't catch stuff using malloc()/free() directly, for example.
4. If leak is due to Python objects holding on to references, rather than due to code not calling free(), tracemalloc or https://pympler.readthedocs.io/en/latest/muppy.html might catch it.

Sticking with Fil

I am happy to accept patches to make Fil work on 32-bit, or perhaps I could do contract work to that end. But that still leaves memory overhead.

I am working on a commercial version of Fil intended to run in production, which reduces overhead (both CPU and memory) by using sampling. So that might help, but 32-bit support is quite low on my list at the moment.

Overall suggestion

Since you have something working, I suggest hacking it some more to just track every Nth allocation instead of all of them, and this may suffice for a one-off memory leak debug session.

[yguel]

Thanks a lot for your detailed answer.

The memory overhead is indeed too much to be able to run the fil-profiler on the device which has specific hardware that we cannot connect simply to a computer with large RAM.

Following your suggestions, we are working now with tracemalloc and it seems that the memory leak is in fact not a memory leak from our code but a problem with fragmented memory blocks allocated by the python garbage collector.