Bug with parallel jobs?

When I run readpst -M -b file.pst it extracts less mail, and a slightly different amount each time, than when it's run with the -j 0 option. An older libpst 0.71 extracts all mail in parallel without -j 0 so maybe there's some concurrency bug that got introduced since then? I can send you pst files if you have trouble reproducing, but for me this happens on basically every .pst file.

[pabs3]

Thanks for the report. Which version/commit of libpst are you running? What is the exact command you are running in the two cases? I'm unable to reproduce this with the sample files I have already. If you could attach a sample without any private data in it that lets you reproduce the issue, that would be good. Are you able to use `git bisect` to track down when this started?

…

-- bye, pabs https://bonedaddy.net/pabs3/

It seems this bug doesn't depend on the version of libpst, but the version of debian. I built the most recent and version 0.6.71 on a debian 11 server and both versions had this behavior while building the same source on debian 10 worked fine. I can reproduce it with the minimal debian 11 container image from https://images.linuxcontainers.org/ and lxc. After:

apt update && apt install wget readpst
wget https://files.enlacehacktivista.org/Pronico%20mail/aBrezhnev.pst
readpst  -D -M -b aBrezhnev.pst
readpst  -D -M -b aBrezhnev.pst
readpst  -D -M -b aBrezhnev.pst
readpst  -j 0 -D -M -b aBrezhnev.pst
du -h --max-depth=1

I get:

419M    ./aBrezhnev
408M    ./aBrezhnev1
417M    ./aBrezhnev2
436M    ./aBrezhnev3

The last one with -j 0 is the largest, while the ones with parallel jobs miss some random amount of mail

[pabs3]

I can reproduce it with the file you linked now.

Thanks for the debugging, that is a very strange bug.

Perhaps it is a compiler or optimsation issue.

I will try to figure out what is going on.

[pabs3]

It definitely isn't affected by the optimisation level. Seems to happen with GCC 9-12 but not GCC 8 on Debian buster.

[pabs3]

I'll continue looking at this tomorrow, but if you have any more ideas for debugging, that would be great.

Yea looks like this could be hard to debug, I "fixed" by just running with -j 0 and xargs -P to parallelize. I tried building with clang on both servers and again it works fine on debian 10, not on 11. Of course it's different versions, llvm-7 vs llvm-11 but it doesn't seem likely that both newer clang and gcc would cause compiler issue. readpst also links a lot of libraries, maybe it's caused by a different version of some library between debian 10 and 11?

[pabs3]

Good point, I'll take a look at this today.

…

-- bye, pabs https://bonedaddy.net/pabs3/

[pabs3]

Looks like you are onto something, building the binary on buster and then copying it to bullseye breaks the parallel extraction even though it still runs fine.

[pabs3]

Upgrading libgsf-1-114 didn't change anything, but upgrading libc6 libgd3 libglib2.0-0 libstdc++6 (and their deps) did. Their dependency versions prevented upgrading them individually.

[pabs3]

I'm doing a debbisect run for this issue of unstable between the release date of buster and the release date of bullseye. This relies on downloads from the Debian wayback machine, which has rate limits, so the run could take a while to complete, but it will pinpoint the exact package difference that will cause the problem. https://manpages.debian.org/man/debbisect https://wiki.debian.org/BisectDebian

…

-- bye, pabs https://bonedaddy.net/pabs3/

[pabs3]

According to the debbisect process, upgrading libc6 from 2.28-10 to 2.29-1 broken the jobs feature.

[pabs3]

I checked this issue on a Fedora 36 live image with glibc 2.35 and found that it still occurs there. I'll discuss it further with the Debian and upstream glibc maintainers.

Seems like a tricky bug, thanks for looking into it!