Exisitng files on mount

[donsony]

Is there some way existing files in a mount can be imported or indexed by OpenCloud?

I have huge amount of data lying in a folder on the same ssd where the OC_DATA_DIR is present. Is it possible I can transfer them to storage folder using SSH and opencloud can index them?

[micbar]

@donsony This is possible in a controlled environment.

OpenCloud can discover files if they belong to the userid 1000. Please make sure they are before you copy them. If you copy them first and change the ownership later, it will not work.

[elderdev]

Hello @micbar . I'm also interested in using files already on my drive. Do I just need to change the permissions on the files and folders and copy them to my OpenCloud user's uploads folder?

[donsony]

I can confirm this works. Rather than using the uploads folder, I used the specific user folder. I found the folder by creating a new file and looking for it inside the file structure

[micbar]

@aduffeck can you answer that?

[aduffeck]

Yes. Just make sure that files can be read and written by the opencloud process (for attaching the metadata) and copy the files into the according space. The new data will then be picked up next time opencloud starts. Alternatively you can run opencloud with STORAGE_USERS_POSIX_WATCH_FS=true in which case it'll use inotify and inotifywait to detect new files automagically.
Please not that this only works with the posix storage driver (which is the default).

[micbar]

@aduffeck FYI

[nearwood]

So, update -- I wasn't patient enough. It looks like all files loaded up:

Perhaps it may have been relevant to mention it was 500 GB of stuff I was expecting. I also moved it to a space rather than a specific user, as that seems more applicable (I migrated from Nextcloud).

[micbar]

Yes. If you move so much data, it takes some time until everything is indexed. OpenCloud assimilates the files during startup.

[aduffeck]

Yes, what @micbar said. Permissions need to be set (and retained) before moving the files into the storage so that opencloud can pick them up on the fly (with collaborative mode STORAGE_USERS_POSIX_WATCH_FS=true).
If you change the permissions afterwards or aren't using collaborative mode, the new files will still be picked up at the initial scan during the next start of opencloud, but that can take a while depending on the storage and the amount of data.

[nearwood]

Ok -- I think I understand now. If you have opencloud running, you have to ensure the permissions are correct before moving so that it doesn't error on or otherwise skip the files before you have a chance to update the permissions. But if you stop opencloud, that shouldn't be a concern, which is what I did. Right?

All is working now! Thanks for your help!

[ccoles146]

Hi, I hope you don't mind me adding information here. I came here with the same question and managed to severely break my OpenCloud install.
I was using NextCloud AIO and decided to move to OpenCloud for my personal/family use as it is lighter weight. I had it working for a few days but when attempting to copy files over managed to break it.

I'm running it as an unprivileged LXC on Proxmox 9 with Docker Compose. I guess it is fairly common for home-use installations.
I have moved all my files from NTFS to a BTRFS drive to be POSIX compliant. I can now either mount it using NFS or a bind mount. The recommended approach is bind mount, so I tried that first. Then of course I wanted to get the UIDs correct, so I used the idmap. Here's what I have:

lxc.idmap: u 0 100000 1000
lxc.idmap: g 0 100000 1000
lxc.idmap: u 1000 1000 1
lxc.idmap: g 1000 1000 1
lxc.idmap: u 1001 101000 64534
lxc.idmap: g 1001 101000 64534

uid and gid 1000 are my user account, chris, on the Proxmox host.

So what happened after this is that the Collabora container would not start up and the logs say:
opencloud-1 | open etc/opencloud/opencloud.yaml: permission denied

So what could have gone wrong with the permissions here? Is it related to 1001 entry which I think is used by Collabora - but from what I understand the mapping is the default for this entry and it worked with default settings.

Thanks in advance.

[ccoles146]

I found a typo, this corrected idmap seems to work.

lxc.idmap: u 0 100000 1000
lxc.idmap: g 0 100000 1000
lxc.idmap: u 1000 1000 1
lxc.idmap: g 1000 1000 1
lxc.idmap: u 1001 101001 64535
lxc.idmap: g 1001 101001 64535

If anyone else is wondering the correct permissions are:

4 drwxr-xr-x  2 1000 1000 4096 Oct 23 14:36 config
4 drwxr-xr-x  3 1000 1000 4096 Oct 23 14:36 data
4 drwxr-xr-x 14 root root 4096 Oct 23 14:35 opencloud-compose

My .env has the following configuration:

OC_CONFIG_DIR=/opt/opencloud/config
OC_DATA_DIR=/opt/opencloud/data