rclone is throwing Errors, restic not working

[schmurian]

I tried creating a backup of my files by mounting internxt via webdav and using rclone.

This throws errors:

2024/12/03 14:52:46 ERROR : data/06/060c225bdaa096d76857123d5724b6317201fe4d7afc349a631a0edacaa3f6b9: Failed to copy: object not found 2024/12/03 14:52:48 ERROR : data/06/06151313327b01f548c8031188fb9da4caf4da5654f5381c602bd56049e0ddc2: Failed to copy: object not found 2024/12/03 14:52:55 ERROR : data/06/06139f2a5377c971a2f9c4355031c1bfaa3c26d02dc798f0cef641dca35c3cf9: Failed to copy: object not found 2024/12/03 14:52:55 ERROR : data/06/061633fa47d57dccd8590d66d66e6988a394413f2834ad535b8cd6c9484775cc: Failed to copy: object not found 2024/12/03 14:53:04 ERROR : data/06/06244debc152f1bd04d387fa7751c6435614b70c61623880f7b893cd0469f7ff: Failed to copy: object not found

restic makes it possible to use rclone from the restic cli, but this already fails at creating the repository key.

CLI version used 1.3.3/ Linux Arm64

[littlehobbyte]

The same issue here.

[szahn1965]

same here with 1.3.3 and 1.4.2

When I make a tar from restic repo I can upload and sync the (splitted) tar-files with rclone/webdav (v1.4.2).

[denissignoretto]

Same issue

[kamiller42]

Same issue. I tried various versions of the initialize repo command. Same result.

Here is my latest version:
restic -o rclone.args="serve restic --stdio --verbose --tpslimit 1 --no-check-certificate --b2-hard-delete" -r rclone:internxt:/restic init

[JanickGers85]

Not using restic but borg via borgmatic that do not have a direct integration with WebDAV, but I use a "post check" bash script tu run sync with rclone to Internxt so I think it should be helpful, so here the results of my labors; there are a few hurdles to make Internxt's WebDAV integration works but somehow I managed to make it.

IMPORANT: I've not tried to check and restore a backup loaded on Internxt, I'll do it when I'll have some time / disk space.
Meawhile I'm using a Hetzner Storage Box for the 3-2-1 backup strategy.

CLI Version used: @internxt/cli/1.4.2 linux-x64 node-v20.12.2
RClone version used: rclone v1.68.2

The rclone command used (BW limit is just for my crappy internet connection, otherwise I won't surf the internet during the sync):

rclone sync {source} {destination} --no-check-certificate -v --bwlimit 1.8M --transfers=1 --create-empty-src-dirs --retries 10

Here a list of error and solutions/workaround.

Timeouts during file upload

This seems to happen when a file upload take "too much time": don't know how much, not misured (should be 15 mins or something like that); let's say it should take max X minutes.
If your internet connection is not able to upload a file in X minutes you'll never sync it on Internxt.
To workaround this I've set parallel transfer to 1 so I'm sure I'll upload the files with the maximum dimension successfully (in my case it's 500MB).
This is another big limitation, for instance I'm not able to sync my parent's backup since their upload bandwidth is just 3Mbit and it takes too much to upload 500MB, and it just fail even with 1 single file upload at time.

Failed to copy: object not found

In my experience this do not means that the file has not been copied: my conclusion is that Internxt's BE is not fast enough to "expose" a file that has been newly created by rclone; in fact if you have retries enabled you'll see that rclone will find the file in the very next retry.
This error happens almost everytime a file has been created, sometimes not but it's very rare.
Here my sync that is running right now:

2025/01/04 14:15:45 ERROR : data/4/4042: Failed to copy: object not found
2025/01/04 14:15:46 ERROR : data/4/4055: Failed to copy: object not found
2025/01/04 14:15:47 ERROR : data/4/4142: Failed to copy: object not found
2025/01/04 14:15:49 ERROR : data/4/4169: Failed to copy: object not found
2025/01/04 14:15:50 ERROR : data/4/4204: Failed to copy: object not found
2025/01/04 14:15:54 ERROR : data/4/4229: Failed to copy: object not found
2025/01/04 14:16:35 INFO  :
Transferred:      541.764 MiB / 482.256 GiB, 0%, 1.718 MiB/s, ETA 3d7h45m
Errors:                29 (retrying may help)
Transferred:            3 / 1077, 0%

Here the files on my account, that has been correctly created

So, if you set a retry on rclone (or leave the default one of 3) you'll get a 0 exit code on the 2nd retry.

Error in creating file in subfolders that don't exists on Internxt (MKCOL-like error during rclone sync)

EDIT: looking at the closed issue seems something related to rclone, solved in the latest version v1.68.2
See this issue: #132

Seems that WebDAV server is not able to create intermediate folders in some circumstances.
For example, this is the directory structure of a Borg repository (in blue the directory; files in numbered directory has not been listed for shortening the list, BTW there's only files in there, no directories)

If you try to sync to an empty folder on Internxt you'll get the error above since seems that the WebDAV server/CLI is not able to create the data directory.
To solve this you have to create the data directory by your own, using the Web App or the mobile app.
But to run successfully a sync you need to accomplish the following point too...

Error 500 (Internal Server Error) when writing to folder created by hand

Honestly I can't imagine why this happens, but to solve this you have to logout and login again with the Internxt's CLI (maybe the token detached and cached during the login process have access only the existing folders and not the manually newly created one? Who knows...).
So you need to internxt logout, then internxt login and then enable again the local WebDAV server as it follows internxt webdav enable

Conclusions

With the steps above I'm able to sync my borg repos, but the issue about the intermediate folder creation can be really limiting if you don't have a strict folder structure to manually create and that don't change in time.

Plus I'm not really comfortable having such a system that needs such tricks to works, I'm asking if it is reliable when I'll really need the backups...

Hope this will help you, since seems that us as paying customers are not supported so much by the vendor, and that's a shame.

[kamiller42]

Thank you for the info, JanickGers85. I upgraded rclone, gave restic very generous rclone args, and same result, "object not found" error.

/ # rclone version
rclone v1.68.2-DEV
- os/version: alpine 3.20.3 (64 bit)
- os/kernel: 6.12.8-0-virt (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.3
- go/linking: dynamic
- go/tags: none

/ # internxt --version
@internxt/cli/1.4.2 linux-x64 node-v22.11.0

/ # restic -o rclone.args="serve restic --stdio --verbose --tpslimit 1 --no-check-certificate --b2-hard-d
elete --retries 20 --low-level-retries 20 --transfers 1" -r rclone:internxt:/restic init

[matt22207]

(UPDATE 2025.03.03 - Duplicati started failing with 404's. Unusable.)

Same. The only backup clients I have been able to get working with internxt (webdav) is Duplicati and rclone copy/sync (standalone, not as backed to other backup clients). Mounting webdav directly through Gnome Nautilus/Files or even through rclone mount throws 500 errors.

Restic also shows:
$ restic -r rclone:internxt:backups-restic init
enter password for new repository: *****
enter password again: *****
rclone: 2025/02/04 19:40:05 ERROR : keys/f28f77e61a9e8e44c2bc816a725d31f4d921943c7ee0d5c0411887caec36d9f6: Post request put error: object not found
rclone: 2025/02/04 19:40:05 ERROR : keys/f28f77e61a9e8e44c2bc816a725d31f4d921943c7ee0d5c0411887caec36d9f6: Post request rcat error: object not found
Fatal: create key in repository at rclone:internxt:backups-restic failed: unexpected HTTP response (500): 500 Internal Server Error

This does result in various folders created on the remote, but generally incomplete init. Same response with the "generous" args in the previous comments.

It seems internxt and/or webdav are not fully compatible.

[schmurian]

@matt22207 what settings do you use for the standalone rclone setup? Does this throw errors too? (for me it did)

[mcdullmedia]

nohup rclone sync /mnt/synohomes internxt:synohomes/current --backup-dir=
internxt:synohomes/archive/"$(date +'%Y-%m-%d_%H-%M-%S')" --create-empty-src-dirs --fast-list &

I have latest rclone docker and the above script seems fine in copy. BTW, the sync just starts for 2 days and its huge. So I am still not sure how sync works. As the internxt webdav seems not support hash, so it only compare file size for sync.

[schmurian]

Thanks, I will try that.

[schmurian]

Ok when running with the -v flag, I still get a Failed to copy: object not found error:

command:
rclone -v sync /path/to/backup/ "internxt:backup" --create-empty-src-dirs --fast-list

2025/02/06 07:35:36 ERROR : snapshots/22713cef8f2d1ff88d7603426f1a4d34e08f611cfbe541b156c5bd0183adc2b7: Failed to copy: object not found
2025/02/06 07:35:51 ERROR : data/06/06139f2a5377c971a2f9c4355031c1bfaa3c26d02dc798f0cef641dca35c3cf9: Failed to copy: object not found
2025/02/06 07:35:52 ERROR : data/06/060c225bdaa096d76857123d5724b6317201fe4d7afc349a631a0edacaa3f6b9: Failed to copy: object not found
2025/02/06 07:35:54 ERROR : data/06/06151313327b01f548c8031188fb9da4caf4da5654f5381c602bd56049e0ddc2: Failed to copy: object not found

rclone version:

rclone v1.60.1-DEV
- os/version: raspbian 12.8 (64 bit)
- os/kernel: 6.6.62+rpt-rpi-2712 (aarch64)
- os/type: linux
- os/arch: arm64
- go/version: go1.19.8
- go/linking: dynamic
- go/tags: none

[mcdullmedia]

Probably need --create-empty-src-dirs so that it will create the folder before putting files

[schmurian]

It's part of the command as you can see. Still didn't work.

[mcdullmedia]

can u try specify a folder inside internxt? As there are pre-created folders at root by default, I suspect sync directly to root may create trouble.

[schmurian]

I've selected a subfolder as part of the initial configuration. So it will try to sync files to the folder backup in the root folder of the internxt webdav. Unfortunately, the same errors occur.