What happens if I use daily replication and the previous replication is taking more than 1 day?

[andrenam]

Hello everyone,
I have a basic question. I want to to use zfs-autobackup to take daily snapshots and replicate them to another system.
Whats happens if for some reason the current replication takes so long, that it is still running when the next cronjob is started?

For example: the initial replication might take long. Can I use a daily cronjob for the initial replication or do I have to do that manually and then setup a cronjob? Or when - for whatever reason - replication has to be reset and begins with full replication? Or when the set of changes is soo big that the replication takes very long?

[andrenam]

in another question, you suggested using flock for that. I'll keep that in mind.
#158

But I'm still curious as to "what would be the result when the second replication is started when the first one is still running". Will this break the replication?

[psy0rz]

It would not really break it, but it might get zfs errors when its trying to send the same dataset at the same time. (dataset busy etc)

And as more jobs start, things will get slower and slower until your system hangs because of memory and IO exhaustion.

Eventually the linux OOM killer will start killing of random jobs, after that the universe implodes or something :P

We could build in automatic locking in zfs-autobackup, but its not that trivial, since you might want to do parallel snapshotting + sending for example.

I'll update the docs with the flock example.

[psy0rz]

You should not run the same cronjob parallel.

So prevent this by using flock or something.

If sometimes a send takes too long but you still want daily snapshots, try splitting it up in a snapshot only cronjob and a send only cronjob.

(This is in the getting started guide)

[psy0rz]

Updated documentation: https://github.com/psy0rz/zfs_autobackup/wiki#avoid-parallel-jobs

[andrenam]

Thank you for your fast and detailed answer. Super.

[no-usernames-left]

take daily snapshots and replicate them to another system

It sounds like https://github.com/jimsalterjrs/sanoid might be exactly what you're looking for.

[psy0rz]

Both tools are designed for this. Are you a bot?😂