--no-cache-sync, endless merging into master chunks index

[grinapo]

(borg version 1.2.4)

There is a system which does fairly rapid backups (eg. hourly) and it is daily pruned, so it's all good.

Except, for a while now daily pruning was blocked by locking (I guess I should use --lock-wait 999999999 or like), so there are a few backups accumulated. Like, 5000+.

Now, the effect is that backup works fine. Nothing else though: an simple info takes about a day (through ssh, if I fire it up locally it finishes in a flash, which is about 50 minutes). Basically anything which needs to collect info on all backups are useless.
(Probably same as #2776 tries to solve since 2017.)

I noticed that borg says that Cache sync: had to fetch 0 B (0 chunks) because no archive had a csize set for them (due to --no-cache-sync).

I am fairly sure I have not used this option anywhere. It seems to be heavily underdocumented (like, does it even exist apart from create, and if not, how to change this behaviour, or, how to make borg not to use it) and I cannot seem to have a way to --yes-cache-sync-please. Since I see that borg takes the time to

Reading cached archive chunk index for myserver-2021-07-06T15:50:02.320726 ...
Verified integrity of /root/.cache/borg/926a1bc19a994ea366828199bd991c270289a5ba7fca7d71ad41f3a61f26fed4/chunks.archive.d/e56361c2c217c8bc06a4ee1c8e7fcebd26dc4d684d6f0c9fe7d6908bce6b06be.compact
Merging into master chunks index ...

[2-8 seconds each, repeat 5000 times]

I suspect that this option may be related to the lack of usability.

(Also at least one prune have finished with no visible pruning, I started a second one, locally, and it's going on for 3 hours now, no output yet apart from

security: saving state for 926a1bc19a994ea366828199bd991c270289a5ba7fca7d71ad41f3a61f26fed4 to /root/.config/borg/security/926a1bc19a994ea366828199bd991c270289a5ba7fca7d71ad41f3a61f26fed4
security: current location   /zdisk/backup/borg/myserver
security: key type           2
security: manifest timestamp 2023-08-21T07:06:55.619132

I wonder if it prunes, I gave --list --stats and hope for some relevant output now.)

[ThomasWaldmann]

I noticed that borg says that Cache sync: had to fetch 0 B (0 chunks)
because no archive had a csize set for them (due to --no-cache-sync).

The cache sync always outputs that (on debug level).

But notice that it says that this was the case for 0 chunks, so it was never the case.

[ThomasWaldmann]

The main issue here is the O(A*F) behaviour with A being the archive count and F being the files count in the archives (assuming that all have a similar files count). Also, the repo being remote contributes to the slowness.

If borg finds that its chunk index is out of sync, it fetches informations for all chunks in all files in each archive to rebuild the main (overall) chunks index (and it also caches information as per archive chunks index so that in case it has to do that again, it would be faster for all the already cached archives).

One often can avoid that the chunks index gets out of sync by not accessing the repo from multiple borg clients.

That's why the docs recommend that (and also not having a very high count of archives).

[grinapo]

The problem is that while I thought I knew this is the theory I realised that borg

• first time creates the per archive chunk cache (takes times), that's expected;
• merges into the main index, sounds great;
• but second time reads all per archive chunk cache and merge again into the main index;
• and every time repeats that, which 5 seconds times few thousand archives equal a real lot of time, every time.

So while it's true that it's faster but it is not much faster, and it does not get any faster somehow (while it, if I understand right, it should have been; but maybe it never saves main idex?).

Also I access the archive from one machine only.

(Well, not right now since I cannot sanely purge the repo through ssh, it takes about 10 hours even locally, but then I'll rebuild the - much smaller - cache on the client and I hope it'll be way faster after.)

[ThomasWaldmann]

The merging of all the per-archive chunk indexes (for all currently existing archives, including new ones, excluding deleted ones) can't be avoided if a overall chunk index shall be created from scratch.

But the remote transfer and some archive metadata parsing is avoided by the local cache.

Once that overall main chunks index is built for all current archives, it should not need rebuilding except if the repo is accessed by another client. Same client means same borg cache (== usually same user, same machine).

Of course, if you would interrupt building of the overall chunks index before it completes, it would try building it again next time.