performance: snapshot find

[enboig]

Sorry, not an issue, but I didn't find a forum to ask.

Right now I use rustic for "server" backups; and kopia for family desktops to a single repository throught a kopia-server.

After searching help, it is not clear if vykar server handle user accounts to limit their access to their files; can you tell me if this is possible?

I am lookig for a tool to allow backing up data from family, and as there are a lot of photos duplicated across computers, duplication is handled across users.

Thanks.

[m3nu]

Alright to ask here.

Currently the server allows concurrent backups, but access can't be limited with different users. And only one repo per server. That's because all the options and authentication apply to the whole server, not per-user or repo.

In the future, I'd like to add access tokens with different permissions.

After that we can look into multi-user and multi-repo on the same server. For now isolating is most secure.

Your idea would be even further out. What you mean is asymetric encryption where users of the same repo can't use each others data. This is the hardest problem in this space. No tool has solved it.

[enboig]

When using client->server backup, I assume:

• I don't trust the client
  • it has its own credentials, not encryption key
  • it cannot delete snapshots
  • It cannot set snapshot date
  • It cannot manipulate snapshot data
• I trust the server:
  • it has the encrytion keys
  • do all repository maintenance, deleting expired snapshots and pruninng repository
  • ensures users only has acces to their own data

Having and admin user who can change snapshots metadata (date, tags, ...) and/or content (deleting huge/confidential files which shouldn't have been backed up) would be a plus.

Note: I have been a backuppc, crashplan, spideroak, duplicati, restic, rustic, kopia user for some years each; and I miss some features of each one of them which I would like to see together someday.

[m3nu]

We don't trust either. So the second item doesn't apply.

Having different permissions is already on the roadmap as per-token permissions.

Currently the append-only setting applies to the whole server. With this change it would apply per-token.

Having used all the tools, what do you still miss in vykar? Now is a good time to influence the direction.

[enboig]

Most modern backup tools are about snapshots to keep consistency (and it is good, an app backup with database and files, should be consistent).

But from a non-techy point of view, it is not. If a document is changed (not destroyed), there should be a way to check versions of a single file across time. Here the old backuppc used to show a table for a folder where you could see files and snapshots versions, so you could see when a file changed, and get any version of it easily (and you could see most updated files of that folder).

Another problem for non-techy users, is what implies forgetting snapshots. If your retention keeps monthly after the first month, any file created and deleted between 1 and 31 is lost forever after the first month. It is hard to understand by them how a file deleted 32 days ago is lost forever, especially when you can restore files from 3 years ago. Here I think a snapshot should keep track of files deleted between previous snapshot and itself (including ones created after previous snapshot).

But this "keep deleted files" option could be problematic on my server, some apps I use have their own backup process which generate a "Ymd.tar.gz" file (not a "backup.tar.gz"), and would get full of deleted files. Here is where configuration should to be able to "opt-in" or "opt-out". Maybee a "retention strategy" profiles could ease it.

[m3nu]

there should be a way to check versions of a single file across time

We already have it. It's vykar snapshot find. You can search for paths and file patterns and it will show you changes of it across all snapshots. I believe we are the only tool offering this. Borgmatic has a wrapper to do it for Borg (I think). More here or in --help: https://vykar.borgbase.com/commands

a file deleted 32 days ago is lost forever

True, this is a limitation of pruning per-snapshot and not per-file. I think this is solved by setting an appropriate prune policy and keep_last and keep_within.

[talios]

I believe we are the only tool offering this.

@m3nu Unless I'm misunderstanding, Kopia supports this:

❯ kopia snapshot ls pom.xml
amrk@minibeard:/Users/amrk/IdeaProjects
  2025-08-01 09:50:00 NZST 8aae64c0bcbf1aa2db0f806419df6289 11.5 KB -rw-r--r-- (monthly-9)
  2025-09-01 11:37:41 NZST 94dd165b46a720c0e193919117cce26a 11.5 KB -rw-r--r-- (monthly-8)
  2025-10-01 08:26:28 NZDT d8beea58050133933a020d9237aacad5 13 KB -rw-r--r-- (monthly-7)
  2025-10-31 15:59:51 NZDT bbbcb7c585002a101e5be5b84b70b6e7 13 KB -rw-r--r-- (monthly-6)
  2025-12-01 09:18:46 NZDT 26a0bf39f69875542fa2abafd5872076 13.9 KB -rw-r--r-- (monthly-5)
  2026-01-01 07:56:23 NZDT c1dc8dc6e02973e15665b3b96885c4c9 14 KB -rw-r--r-- (monthly-4,annual-2)
  2026-02-01 11:29:00 NZDT 7b7c69e4d9faa578227175516e4d6bb8 14 KB -rw-r--r-- (monthly-3)
  2026-02-22 21:24:32 NZDT a5ec7bef76b32125927cc1c82fc34a61 14 KB -rw-r--r-- (latest-9..10,daily-6..7,weekly-2..4,monthly-2)
  + 4 identical snapshots until 2026-03-09 09:46:04 NZDT
  2026-03-10 08:24:13 NZDT c1dc8dc6e02973e15665b3b96885c4c9 14 KB -rw-r--r-- (latest-1..8,hourly-1..4,daily-1..5,weekly-1,monthly-1,annual-1)
  + 7 identical snapshots until 2026-03-14 10:50:04 NZDT

Here I'm lsing for all versions of pom.xml in my current directory, and Kopia lists all the snapshots, and their floating tags (monthly-4, annual-2 etc.)

Then restoring from one of the snapshots is simple:

kopia snapshot restore c1dc8dc6e02973e15665b3b96885c4c9 pom.xml

If I'd deleted that file a while ago, it wouldn't show the last latest-1..10 snapshots, but likely just the monthly-3 snapshot, which I could restore from.

As you say tho, setting a keep_last etc. policy would likely help here.

Interestingly, I just tried vykar snapshot find pom.xml in the same directory to compare the results, 10+ minutes, and it still hasn't returned, so I'm not sure if vykar is searching thru every file matching pom.xml, or just those in the directory I'm currently in. Altho I see I need to use vykar snapshot find --name pom.xml . (unfortunately, that also hasn't returned in 10+ minutes - and I note it's also not prompting me to access 1password like vykar list etc. does (my passcommand).

[m3nu]

Should work similar to Kopia. 10 minutes is a bit long though. Should be seconds. What's your backend? This was from the REST backend:

vykar snapshot find --iname 'wordpress-domains.txt'
Documents/wordpress-domains.txt
  Snapshot     Date                       Size  Status
  e0ea376d     2026-03-13 21:08:53       705 B  added
  9cdc6f48     2026-03-13 21:28:41       705 B  unchanged
  e1dddf17     2026-03-13 21:30:51       705 B  unchanged
  9ca0c7de     2026-03-13 21:31:47       705 B  unchanged
  f65b49aa     2026-03-13 21:36:02       705 B  unchanged

[talios]

@m3nu Backblaze B2 (not using the server, so not really related to this ticket).

[m3nu]

All backends should work the same and the REST server works very similar to the S3 backend. Currently S3 is only tested on minio and garage locally.

[talios]

At 10 minutes, I ctrl-c'd the process as I was heading out. I'm running it again now and just leaving it. I'll raise this as a separate issue, so we don't take over this issue.

[m3nu]

No need. Already found the issue: We don't group reads on the same pack together properly for this feature. It's already done for restore, but not find. Will have a commit to fix this up shortly. This should speed up most backends, but likely S3 and SFTP the most. For REST and local it's less notable.

[talios]

@m3nu awesome! You rock.

[m3nu]

Done. This brings search time from 1m to 6s for REST. Should be faster on S3 too, but don't have it set up right now.

./vykar snapshot find --iname 'wordpress-domains.txt'  1.10s user 0.47s system 2% cpu 1:03.42 total
./vykar snapshot find --iname 'wordpress-domains.txt'  0.80s user 0.13s system 15% cpu 6.063 total

Fixed in 9945351

[talios]

Testing against Backblaze b2 from here:

$ time ./target/release/vykar snapshot find --iname pom.xml upstream/sisu-project/org.eclipse.sisu.plexus/src/test/resources/component-jar
....
Executed in  172.84 secs    fish           external
   usr time    8.45 secs    0.51 millis    8.45 secs
   sys time    3.55 secs    2.17 millis    3.55 secs

[m3nu]

3 minutes? Could still be faster. When using vykar -vv snapshot find ... do you see specific requests that stand out? I.e. too many of a specific kind.