Huge stack usage and valgrind thread errors

[RoxtroffJ]

Hello,

I am currently working on a project that uses the flow5 library, and while debugging, I found out that the library uses a suspiciously large amount of stack, and triggers valgrind drd. Indeed, this code:

#include <iostream>

#include <api.h>
#include <foil.h>

int main()
{
    printf("test threaded naca\n");


    Foil *pFoilN2413 = foil::makeNacaFoil(2413, "NACA 2413");
    

    globals::deleteObjects();

    std::cout << "done" << std::endl;
    return 0;
}

is enough to produce more than 1000 valgrind drd errors, and has one thread using 3.2MB of stack:

$ valgrind --tool=drd --show-stack-usage=yes ./Main

outputs this (here I interrupted quite quickly, otherwise it takes ages, and ends with valgrind giving up at the 1000th error)

==71565== drd, a thread error detector
==71565== Copyright (C) 2006-2020, and GNU GPL'd, by Bart Van Assche.
==71565== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==71565== Command: ./Main
==71565== 
test threaded naca
==71565== Conflicting store by thread 1 at 0x079ecf80 size 8
==71565==    at 0x59B5093: exec_blas_async (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x59BA298: dgetrf_parallel (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x59B9C88: dgetrf_parallel (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x59B9C88: dgetrf_parallel (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x59B9C88: dgetrf_parallel (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x59B9C88: dgetrf_parallel (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x577DA08: dgetrf_ (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x49D2EB7: matrix::solveLinearSystem(int, double*, int, double*, int) (in /usr/local/lib/libfl5-lib.so.1.0.0)
==71565==    by 0x4994119: CubicSpline::solve() (in /usr/local/lib/libfl5-lib.so.1.0.0)
==71565==    by 0x4993FF0: CubicSpline::updateSpline() (in /usr/local/lib/libfl5-lib.so.1.0.0)
==71565==    by 0x499600E: CubicSpline::approximate(int, std::vector<Node2d, std::allocator<Node2d> > const&) (in /usr/local/lib/libfl5-lib.so.1.0.0)
==71565==    by 0x49E6F0B: Foil::initGeometry() (in /usr/local/lib/libfl5-lib.so.1.0.0)
==71565== Allocation context: BSS section of /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so
==71565== Other segment start (thread 2)
==71565==    at 0x48602FF: pthread_mutex_unlock@* (in /usr/libexec/valgrind/vgpreload_drd-amd64-linux.so)
==71565==    by 0x59B3F90: blas_memory_alloc (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x59B4A75: ??? (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x48558B8: ??? (in /usr/libexec/valgrind/vgpreload_drd-amd64-linux.so)
==71565==    by 0x5457AA3: start_thread (pthread_create.c:447)
==71565==    by 0x54E4A63: clone (clone.S:100)
==71565== Other segment end (thread 2)
==71565==    at 0x54C980B: sched_yield (syscall-template.S:120)
==71565==    by 0x59B4AB9: ??? (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x48558B8: ??? (in /usr/libexec/valgrind/vgpreload_drd-amd64-linux.so)
==71565==    by 0x5457AA3: start_thread (pthread_create.c:447)
==71565==    by 0x54E4A63: clone (clone.S:100)
==71565== 
^C==71565== 
==71565== Process terminating with default action of signal 2 (SIGINT)
==71565==    at 0x59BA320: dgetrf_parallel (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x59B9C88: dgetrf_parallel (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x59B9C88: dgetrf_parallel (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x59B9C88: dgetrf_parallel (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x59B9C88: dgetrf_parallel (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x577DA08: dgetrf_ (in /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so)
==71565==    by 0x49D2EB7: matrix::solveLinearSystem(int, double*, int, double*, int) (in /usr/local/lib/libfl5-lib.so.1.0.0)
==71565==    by 0x4994119: CubicSpline::solve() (in /usr/local/lib/libfl5-lib.so.1.0.0)
==71565==    by 0x4993FF0: CubicSpline::updateSpline() (in /usr/local/lib/libfl5-lib.so.1.0.0)
==71565==    by 0x499600E: CubicSpline::approximate(int, std::vector<Node2d, std::allocator<Node2d> > const&) (in /usr/local/lib/libfl5-lib.so.1.0.0)
==71565==    by 0x49E6F0B: Foil::initGeometry() (in /usr/local/lib/libfl5-lib.so.1.0.0)
==71565==    by 0x4A091FA: Objects2d::makeNacaFoil(Foil*, int, int) (in /usr/local/lib/libfl5-lib.so.1.0.0)
==71565== thread 8 finished and used 2679 bytes out of 8323072 on its stack. Margin: 8320393 bytes.
==71565== thread 4 finished and used 2679 bytes out of 8323072 on its stack. Margin: 8320393 bytes.
==71565== thread 6 finished and used 2679 bytes out of 8323072 on its stack. Margin: 8320393 bytes.
==71565== thread 2 finished and used 2775 bytes out of 8323072 on its stack. Margin: 8320297 bytes.
==71565== thread 7 finished and used 2679 bytes out of 8323072 on its stack. Margin: 8320393 bytes.
==71565== thread 5 finished and used 2679 bytes out of 8323072 on its stack. Margin: 8320393 bytes.
==71565== thread 3 finished and used 2679 bytes out of 8323072 on its stack. Margin: 8320393 bytes.
==71565== thread 1 finished and used 3254991 bytes out of 8388608 on its stack. Margin: 5133617 bytes.
==71565== 
==71565== For lists of detected and suppressed errors, rerun with: -s
==71565== ERROR SUMMARY: 5 errors from 1 contexts (suppressed: 248 from 105)

So I was wandering if this is normal or if openblas should be used single threaded, or if something else is going on here.

Thanks in advance for your help!

[techwinder]

Not sure. I don't really understand the error message. Can't see anything obviously wrong on the flow5 side of the code, nor any abnormal stack memory usage.
I wouldn't be surprised if OpenBlas made intensive use of the stack memory rather than the heap because it's faster to access.
To eliminate the foil & spline related code, you could try to run valgrind on a simple call to void testLapacke() located in the testmatrix.cpp file.

[RoxtroffJ]

Hello, thank you for the response.

testLapacke() appears to work fine (no valgrind errors and 13kB of stack usage).

I also did more testing on the makeNacaFoil call, setting the OPENBLAS_NUM_THREADS=1 env variable makes it run twice as fast, with no Valgrind issues, and 41kB of stack used.

But of course it might hurt the performance for more complex operations...

[techwinder]

Did some tests on my side with the same valgrind error message. No error message on the other hand when fl5-lib is linked to the single-threaded version of OpenBlas.
I also get the same type of error messages when fl5-lib is linked to MKL.
My best guess is that it's a conflict with the multi-threading of the Qt framework used by flow5. But it may be beyond my programming skills to fix.

[RoxtroffJ]

Alright, if it is indeed that, then it is also beyond my programming skills.

Nevertheless, thank you for your help! :)