Restore Performance

[kalleyne]

This is a placeholder.

Original thread: https://forum.restic.net/t/restic-restore-showing-a-sharp-reduction-of-rx-receive-speed-over-time/6199

[aawsome]

@kalleyne Thanks for opening this discussion.

Actually I don't know if I can confirm that CPU usage generally is a bottleneck when restoring.
Just did a test with a 1,5 GB file generated from /dev/urandom. Backed up to a repo on tmpfs and restored to tmpfs.

Using rustic, restore was <3s and reported >600 MiB/s (#624 even increases restore rate).
This used ~500% CPU, i.e. 5 cores in my Intel Core i7-8650U Laptop.
It could be that in this scenario CPU is the bottleneck, but we are talking about way more than 1Gbit/s...

For comparison:
restic restore was <9s, but used less CPU. Most likely the restic restore is less parallelized than rustics. So, with restic, CPU and few parallelization could be a bottleneck.

[kalleyne]

Hello @aawsome

I think that our experience during the roughly week long restore somewhat matches the results you see with your test.

Our impression is that LAN based disk I/O is probably the first bottleneck that users may encounter with restic or rustic restores. The performance of SSDs and NVMe disks are in most cases better than spinning disks. Still though, I think it's safe to state that rustic restore read and write requests are not saturating the disks within the ZFS pool either at the source or target side.

Based upon some quick estimates from MinIO for example:

https://min.io/docs/minio/linux/operations/checklists/hardware.html

Networking has the greatest impact on MinIO performance, where low per-host bandwidth artificially constrains the potential performance of the storage. The following examples of network throughput constraints assume spinning disks with ~100MB/S sustained I/O

1GbE network link can support up to 125MB/s, or one spinning disk

10GbE network can support approximately 1.25GB/s, potentially supporting 10-12 spinning disk

25GbE network can support approximately 3.125GB/s, potentially supporting ~30 disks

Seeing that the rustic restore that we ran had 16 spinning disks on both the source and target and we were getting throughput of less than 1GbE (though resilvering on source) then we hoped that we could find some measure of improvement somehow if a bottleneck can be addressed somewhere.

But, yes, we are sure that there could be a next level bottleneck at the CPU level if disk I/O is at the speed of virtual memory or RAM disk and if we wanted to push restore performance to exceed throughput of 5-10Gbit/s....which would be desirable.

Unfortunately for many of our customers there seemed to be a lot of pre-conditioning going on where rsync has been equated to backups. Multiple simultaneous rsync jobs done over SMB or even SSH can get close to saturating 10GbE. So while restic/rustic restores are doing a lot of CPU bound work and while restic/rustic repos bring to the table a huge list of positive qualities....most people are not comparing apples with apples and will only look at the final result and declare that "multiple rsync restores are faster" than any single use of restic/rustic restore.

So yes, training and preparing staff is part of the answer. But we are not choosy and we will gladly accept any rustic performance improvements open for consideration.

Thanks.

[wscott]

My assumption is that you are thrashing the spinning disk with multiple threads writing parallel streams of data. It is crazy fast on NVMe, but with disks that need to seek the parallel threads is killing performance. Can you configure rustic (or restic) to not parallelize the restore and see how that changes things?

[aawsome]

I think I can prepare a rustic version were

• parallel reads from the repo
• cpu cores used to decrypt/uncompress
• parallel writes to the "local" dir

can be customized + where we get debugging information which part is waiting for which other part in this processing pipeline. But in order to do that I need some time, so this could take a couple of days.

About the comment of @wscott: In fact the situation is even worse: rustic does quite randomly pick some chunks from the backend, decrypts and decompresses them and then writes them to every destination where this chunk is used. That is, the data is very randomly+parallel written to the restore destination. I also thought to bring this random access into some order such that the probability of writing sequentially to a file on the destination increases a lot. Actually #624 is some preliminary work for that. But anyway I think we start with optimizing the throughput (which includes finetuning the number of parallel writes) and then do some optimization of local writes in a second step.