Using zfs diff to directly find files for incremental backup with no scanning

[infogulch]

If the backup source is on ZFS, borg could use the zfs diff command to directly list out files that have changed since the last snapshot and how (create,delete,modify,rename) in order to avoid scanning the whole volume.

Example output:

# zfs diff -F tank/test@before tank/test
M       /       /tank/test/
M       F       /tank/test/linked      (+1)
R       F       /tank/test/oldname -> /tank/test/newname
-       F       /tank/test/deleted
+       F       /tank/test/created
M       F       /tank/test/modified

I'm imagining it working like this:

1. Take snapshot-1, do first (full) backup from snapshot-1 (backing up from a snapshot avoids inconsistent state backups, which is a different benefit)
2. Take snapshot-2, do incremental backup by running zfs diff snapshot-1 snapshot-2 and backing up the listed differences out of snapshot-2. when completed delete snapshot-1.
3. Take snapshot-3, do incremental backup by running zfs diff snapshot-2 snapshot-3 and backing up the listed differences out of snapshot-3. when completed delete snapshot-2.
4. (etc)

Some thoughts:

• These should be "borg-managed snapshots" and probably labeled as such along with the backup id so users don't mess with them
• The downside of keeping a snapshot of the previous backup is that it can increase storage space usage while the snapshot is maintained. This can be mitigated in a few ways:
  • Only keep one previous snapshot. This bounds the maximum additional space used to the incremental difference between the snapshot and the current state. The minimum additional space could be as low as zero depending on the workload, which leads to....
  • Be smart about which workloads to use this strategy on. This would work great if there are many more files left unmodified than changed, and even more so if most changes are creates.
    • Good uses might include append only data stores with many files like git objects, or historical archives that are only written to like files generated by ArchiveBox, or object store backends like minio where its workload has a favorable profile (attachments)
    • Bad uses would be things like databases: very few files that constantly change, comparing snapshots doesn't provide any benefit, and you get all the downsides of keeping a snapshot that increases disk usage
• "Just use ZFS Send" - while ZFS send is pretty ideal if you have a full computer that runs just to hold backups, it won't work if your destination is simple storage (s3/ssh), and borg has some other nice benefits for backup use-cases which I'm sure I don't need to sell here :)

What do you think?

[ThomasWaldmann]

Well, guess we had that "find filenames of modified files and give them to borg to optimize" idea already in some other ticket.

In short: somehow unexpectedly, this is not useful, as borg creates full backups including all fs items.

So it needs to visit every fs item anyway and it also needs to read some file metadata from each anyway (like acls, xattrs).
When doing that, it can also quickly find out if the item has been modified and if not, skip reading/hashing/archiving the file content data.

[ThomasWaldmann]

borg does not read old archives to use them as a base for creating a new archive, but gets all from:

• the fs (metadata)
• the files cache (content chunk ids, IF the file contents were not changed)

theoretically, reading some previous archive for that would be thinkable, but things might get complicated as borg does only have sequential access to data in the archive, not random access.

[infogulch]

borg only has sequential access to data in the archive, not random access

That's interesting and not what I would expect, why is that? Because borg is optimized for writing to tape?

[ThomasWaldmann]

No, it is not intended for tape. But a borg archive (which contains the metadata stream and references to data chunks) is quite similar to a tar archive. The chunks need random access though.

[infogulch]

Oh that makes sense.

Would random access make it more efficient? If there were a compact representation of offsets within the archive would that help?

Is there a document describing the on-disk format?

[ThomasWaldmann]

Check if the "internals" docs have something. Transforming an archive to be indexed and random access would be a major change and is currently out of question I think.