diff --git a/app/dcs/getting-started/page.md b/app/dcs/getting-started/page.md index 7953c408f..6274cc1ec 100644 --- a/app/dcs/getting-started/page.md +++ b/app/dcs/getting-started/page.md @@ -171,14 +171,23 @@ Next we'll upload a file. Here is an example image of a tree growing hard drives Copy the file to your bucket. -{% code-group %} +{% tabs %} +{% tab label="rclone" %} + +{% callout type="info" %} +Using `--disable-http2` with rclone for Storj is recommended for increased transfer speeds by avoiding HTTP/2 specific issues. +{% /callout %} ```shell {% title="rclone" %} # link[8:11] https://rclone.org/commands/rclone_copy/ # terminal -rclone copy ~/Downloads/storj-tree.png storj:my-bucket/ +rclone copy --disable-http2 ~/Downloads/storj-tree.png storj:my-bucket/ ``` +{% /tab %} + +{% tab label="aws cli" %} + ```shell {% title="aws cli" %} # focus # terminal @@ -187,26 +196,39 @@ aws s3 --endpoint-url=https://gateway.storjshare.io cp ~/Downloads/storj-tree.pn upload: Downloads/storj-tree.png to s3://my-bucket/storj-tree.png ``` -{% /code-group %} +{% /tab %} + +{% /tabs %} ## Download file To retrieve the file, use the same command as upload but reverse the order of the arguments -{% code-group %} +{% tabs %} +{% tab label="rclone" %} + +{% callout type="info" %} +Using `--disable-http2` with rclone for Storj is recommended for increased transfer speeds by avoiding HTTP/2 specific issues. +{% /callout %} ```shell {% title="rclone" %} # terminal # link[8:11] https://rclone.org/commands/rclone_copy/ -rclone copy storj:my-bucket/ ~/Downloads/storj-tree-2.png +rclone copy --disable-http2 storj:my-bucket/ ~/Downloads/storj-tree-2.png ``` +{% /tab %} + +{% tab label="aws cli" %} + ```shell {% title="aws cli" %} # terminal aws s3 --endpoint-url=https://gateway.storjshare.io cp s3://my-bucket/ ~/Downloads/storj-tree-2.png ``` -{% /code-group %} +{% /tab %} + +{% /tabs %} ## List files diff --git a/app/dcs/third-party-tools/rclone/page.md b/app/dcs/third-party-tools/rclone/page.md index 0ee14b1d9..b9b4cb1aa 100644 --- a/app/dcs/third-party-tools/rclone/page.md +++ b/app/dcs/third-party-tools/rclone/page.md @@ -13,56 +13,26 @@ metadata: different integration patterns for optimized performance and security. --- -## Introduction +Follow the [Getting Started guide](docId:AsyYcUJFbO1JI8-Tu8tW3) to setup Rclone. -These 'Getting Starting' tutorials will showcase the process for configuring [Rclone](https://rclone.org) with Storj DCS. We will cover only some of the basic features in this guide. +The following is more details about the 2 ways you can use Rclone with Storj. -To make the most of Rclone, look at the complete Rclone [command reference](https://rclone.org/commands/). +## S3 Compatible -{% callout type="danger" %} -These guides are experimental. The main functionality appears to work, but there are expected to be undiscovered issues (including issues around connection timeouts). Please report any issues you may run into on this [forum thread](https://forum.storj.io/t/two-more-tech-previews-rclone-and-restic/6072). -{% /callout %} +Use our [S3 compatible API](docId:eZ4caegh9queuQuaazoo) to increase upload performance and reduce the load on your systems and network. A 1GB upload will result in only 1GB of data being uploaded -You will need one of the following: - -- [](docId:Ch4vLynsEqyT2-3qDEBiy) that someone else shared with you, or - -- [](docId:OXSINcFRuVMBacPvswwNU) (access token) of a Storj DCS project you are a member of. - -## Selecting an Integration Pattern +- Faster upload +- Reduction in network load +- Server-side encryption ## Native Use our native integration pattern to take advantage of client-side encryption as well as to achieve the best possible download performance. Uploads will be erasure-coded [](docId:Pksf8d0TCLY2tBgXeT18d), thus a 1GB upload will result in 2.68GB of data being uploaded to storage nodes across the network. -### Use this pattern for - -- The strongest security - -- The best download speed - -{% quick-links %} -{% quick-link %} -[](docId:Mk51zylAE6xmqP7jUYAuX) -{% /quick-link %} -{% /quick-links %} - -## Hosted Gateway - -Use our S3 compatible Hosted Gateway integration pattern to increase upload performance and reduce the load on your systems and network. Uploads will be encrypted and erasure-coded [](docId:hf2uumViqYvS1oq8TYbeW), thus, a 1GB upload will result in only 1GB of data being uploaded to the Hosted Gateway. - -### Use this pattern for - -- Reduced upload time - -- Reduction in network load - -{% callout type="info" %} -By selecting this integration pattern, you are opting in to server-side encryption. -{% /callout %} +- End-to-end encryption +- Faster download speed {% quick-links %} -{% quick-link %} -[](docId:WayQo-4CZXkITaHiGeQF_) -{% /quick-link %} +{% quick-link title="Rclone S3 compatible" href="docId:AsyYcUJFbO1JI8-Tu8tW3" /%} +{% quick-link title="Rclone native" href="docId:Mk51zylAE6xmqP7jUYAuX" /%} {% /quick-links %} diff --git a/app/dcs/third-party-tools/rclone/rclone-native/page.md b/app/dcs/third-party-tools/rclone/rclone-native/page.md index 47f5a71cd..18806363d 100644 --- a/app/dcs/third-party-tools/rclone/rclone-native/page.md +++ b/app/dcs/third-party-tools/rclone/rclone-native/page.md @@ -18,7 +18,6 @@ Use our native integration pattern to take advantage of client-side encryption a ## Use this pattern for - The strongest security - - The best download speeds ## Setup @@ -180,158 +179,4 @@ q) Quit config e/n/d/r/c/s/q> q ``` -{% callout type="info" %} -For additional security, you should consider using the (s) option - -`Set configuration password` option. It will encrypt the `rclone.conf` configuration file. This way secrets like the [](docId:OXSINcFRuVMBacPvswwNU) (access token), the encryption passphrase, and the access grant won't be stolen if an attacker get access to your configuration file. -{% /callout %} - -## Create a Bucket - -Use the `mkdir` command to create new bucket, e.g. `mybucket`. - -```yaml -rclone mkdir waterbear:mybucket -``` - -## List All Buckets - -Use the `lsf` command to list all buckets. - -```yaml -rclone lsf waterbear: -``` - -{% callout type="info" %} -Note the colon (`:`) character at the end of the command line. -{% /callout %} - -## Delete a Bucket - -Use the `rmdir` command to delete an empty bucket. - -```yaml -rclone rmdir waterbear:mybucket -``` - -Use the `purge` command to delete a non-empty bucket with all its content. - -```yaml -rclone purge waterbear:mybucket -``` - -## Upload Objects - -Use the `copy` command to upload an object. - -```yaml -rclone copy --progress ~/Videos/myvideo.mp4 waterbear:mybucket/videos/ -``` - -{% callout type="info" %} -The `--progress` flag is for displaying progress information. Remove it if you don't need this information. -{% /callout %} - -Use a folder in the local path to upload all its objects. - -```yaml -rclone copy --progress ~/Videos/ waterbear:mybucket/videos/ -``` - -{% callout type="info" %} -Only modified files will be copied. -{% /callout %} - -## List Objects - -Use the `ls` command to list recursively all objects in a bucket. - -```yaml -rclone ls waterbear:mybucket -``` - -Add the folder to the remote path to list recursively all objects in this folder. - -```yaml -rclone ls waterbear:mybucket/videos/ -``` - -Use the `lsf` command to list non-recursively all objects in a bucket or a folder. - -```yaml -rclone lsf waterbear:mybucket/videos/ -``` - -## Download Objects - -Use the `copy` command to download an object. - -```yaml -rclone copy --progress waterbear:mybucket/videos/myvideo.mp4 ~/Downloads/ -``` - -{% callout type="info" %} -The `--progress` flag is for displaying progress information. Remove it if you don't need this information. -{% /callout %} - -Use a folder in the remote path to download all its objects. - -```yaml -rclone copy --progress waterbear:mybucket/videos/ ~/Downloads/ -``` - -## Delete Objects - -Use the `deletefile` command to delete a single object. - -```yaml -rclone deletefile waterbear:mybucket/videos/myvideo.mp4 -``` - -Use the `delete` command to delete all object in a folder. - -```yaml -rclone delete waterbear:mybucket/videos/ -``` - -## Print the Total Size of Objects - -Use the `size` command to print the total size of objects in a bucket or a folder. - -```yaml -rclone size waterbear:mybucket/videos/ -``` - -## Sync Two Locations - -Use the `sync` command to sync the source to the destination, changing the destination only. Doesn’t transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary. - -```yaml -rclone sync --progress ~/Videos/ waterbear:mybucket/videos/ -``` - -{% callout type="info" %} -The `--progress` flag is for displaying progress information. Remove it if you don't need this information. -{% /callout %} - -{% callout type="info" %} -Since this can cause data loss, test first with the `--dry-run` flag to see exactly what would be copied and deleted. -{% /callout %} - -The sync can be done also from Storj DCS to the local file system. - -```yaml -rclone sync --progress waterbear:mybucket/videos/ ~/Videos/ -``` - -Or between two Storj DCS buckets. - -```yaml -rclone sync --progress waterbear-us:mybucket/videos/ waterbear-europe:mybucket/videos/ -``` - -Or even between another cloud storage and Storj DCS. - -```yaml -rclone sync --progress s3:mybucket/videos/ waterbear:mybucket/videos/ -``` +For additional commands you can do, see [](docId:WayQo-4CZXkITaHiGeQF_). diff --git a/app/dcs/third-party-tools/rclone/rclone-s3/page.md b/app/dcs/third-party-tools/rclone/rclone-s3/page.md index 1e6a0006e..841d46a4c 100644 --- a/app/dcs/third-party-tools/rclone/rclone-s3/page.md +++ b/app/dcs/third-party-tools/rclone/rclone-s3/page.md @@ -1,5 +1,5 @@ --- -title: Rclone S3 API +title: Rclone additional commands docId: WayQo-4CZXkITaHiGeQF_ redirects: - /dcs/how-tos/sync-files-with-rclone/rclone-with-hosted-gateway @@ -8,211 +8,15 @@ metadata: description: Step-by-step guide to configure Rclone pointed to Storj's S3 compatible API, providing better upload performance and lower network load. --- -## Selecting an Integration Pattern - -Use our S3-compatible Hosted Gateway integration pattern to increase upload performance and reduce the load on your systems and network. Uploads will be encrypted and erasure-coded server-side; thus, a 1GB upload will result in only 1GB of data being uploaded from the client to the hosted gateway. - -## Use this pattern for - -- Reduced upload time - -- Reduction in network load - {% callout type="info" %} -By selecting this integration pattern, you are opting into [Server-side Encryption](docId:hf2uumViqYvS1oq8TYbeW). -{% /callout %} - -## Prerequisites - -### Generate Credentials to the Gateway MT - -**Navigate to the Access** page within your project and then click on **Create S3 Credentials**. A modal window will pop up where you should enter a name for this access grant. - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/U_P56dlNYzj-p7I4Ubsvj_rclone1.png) - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/xTdExe6AA-ZbmJWOqNmSf_rclone2.png) - -**Assign the permissions** you want this access grant to have, then click on **Encrypt My Access**: - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/m_NwIW3B7Rx5xOL1zRAwz_rclone3.png) - -**Enter the Encryption Passphrase** you used for your other access grants. If this is your first access grant, we strongly encourage you to use a mnemonic phrase as your encryption passphrase. (The GUI automatically generates one on the client-side for you) - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/JrZT5rCAHWkwTWMy-iJzE_rclone4.png) - -{% callout type="warning" %} -**This passphrase is important!** Encryption keys derived from it are used to encrypt your data at rest, and your data will have to be re-uploaded if you want it to change! - -Importantly, if you want two access grants to have access to the same data, **they must use the same passphrase**. You won’t be able to access your data if the passphrase in your access grant is different than the passphrase you uploaded the data with. - -Please note that **Storj does not know or store your encryption passphrase**, so if you lose it, you will not be able to recover your files. -{% /callout %} - -Click either on the **Copy to clipboard** link or **Download .txt** and then confirm that you copied your Encryption Phrase to a safe place. - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/1tsIfAbcVWQWViVWNSYF1_rclone5.png) - -Click the **Create my Access** link to finish generating of S3 credentials. - -![](https://link.storjshare.io/raw/jua7rls6hkx5556qfcmhrqed2tfa/docs/images/m4gl1YLwvpUBQ0DTu6mQe_rclone6.png) - -Copy your **Access Key**, **Secret Key**, and **Endpoint** to a safe location or download them. - -Now you are ready to **configure** Rclone - -## Setup - -First, [Download](https://rclone.org/downloads/) and extract the rclone binary onto your system. - -Execute the config command: - -```bash -rclone config -``` - -A text-based menu will prompt. Type `n` and hit `Enter` to create a new remote configuration, select n (New Remote). - -```Text -e) Edit existing remote -n) New remote -d) Delete remote -r) Rename remote -c) Copy remote -s) Set configuration password -q) Quit config -e/n/d/r/c/s/q> n -``` - -Enter a name for the new remote configuration, e.g., `waterbear`. - -```none -name> waterbear -``` - -A long list of supported storage backends will prompt. Enter `s3` and hit `Enter`. - -```none -Option Storage. -Type of storage to configure. -Choose a number from below, or type in your own value. -... -5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi - \ (s3) -... -Storage> s3 -``` - -A further list of S3 storage providers will prompt. Enter `Storj` and hit `Enter`. - -```none -Option provider. -Choose your S3 provider. -Choose a number from below, or type in your own value. -Press Enter to leave empty. -... -21 / Storj (S3 Compatible Gateway) - \ (Storj) -... -provider> Storj -``` - -A choice will be given on how you will enter credentials. Hit `Enter` for the default choice of `1 / Enter AWS credentials in the next step`. - -```none -1 / Enter AWS credentials in the next step. -``` - -You will be asked for your Access Key ID followed by the Secret Access Key that you previously generated, follow the pattern in the code block below. - -```none -Option access_key_id. -AWS Access Key ID. -Leave blank for anonymous access or runtime credentials. -Enter a value. Press Enter to leave empty. -access_key_id> your-access-key-id - -Option secret_access_key. -AWS Secret Access Key (password). -Leave blank for anonymous access or runtime credentials. -Enter a value. Press Enter to leave empty. -secret_access_key> your-secret-access-key -``` - -You will be asked what Endpoint to connect to. Enter `1` to select the Global Hosted Gateway and hit `Enter`. - -{% callout type="warning" %} -If you use an older Rclone release, you will see a list of regional gateway endpoints to select from. Ignore them and enter the global `gateway.storjshare.io` endpoint. Using the regional endpoints may cause performance issues. +Follow the [Getting Started guide](docId:AsyYcUJFbO1JI8-Tu8tW3) to setup Rclone. {% /callout %} -```dockerfile -Option endpoint. -Endpoint of the Shared Gateway. -Choose a number from below, or type in your own value. -Press Enter to leave empty. - 1 / Global Hosted Gateway - \ (gateway.storjshare.io) -endpoint> 1 -``` +The follow are additional commands or options you can consider when using Rclone -You will be asked if you want to edit the advanced config. Type `y` and hit `Enter`. Skip most of the questions by hitting `Enter`. The important ones to configure are `chunk_size` (64M) and `disable_checksum` (true). - -```none -Edit advanced config? -y) Yes -n) No (default) -y/n> y -... -Option chunk_size. -Chunk size to use for uploading. -When uploading files larger than upload_cutoff or files with unknown -size (e.g. from "rclone rcat" or uploaded with "rclone mount" or google -photos or google docs) they will be uploaded as multipart uploads -using this chunk size. -... -Enter a size with suffix K,M,G,T. Press Enter for the default (5Mi). -chunk_size> 64M -... -Option disable_checksum. -Don't store MD5 checksum with object metadata. -Normally rclone will calculate the MD5 checksum of the input before -uploading it so it can add it to metadata on the object. This is great -for data integrity checking but can cause long delays for large files -to start uploading. -Enter a boolean value (true or false). Press Enter for the default (false). -disable_checksum> true -``` +## Configuration password -A summary of the remote configuration will prompt. Type `y`and hit `Enter` to confirm. - -```none -[waterbear] -type = s3 -provider = Storj -access_key_id = -secret_access_key = -endpoint = gateway.storjshare.io -chunk_size = 64Mi -disable_checksum: true --------------------- -y) Yes this is OK (default) -e) Edit this remote -d) Delete this remote -y/e/d> y -``` - -Now you should see one remote configuration available. Enter `q` and hit `Enter` to quit the configuration wizard. - -```none -Current remotes: - -Name Type -==== ==== -waterbear s3 -``` - -{% callout type="info" %} -For additional security, you should consider using the `s) Set configuration password` option. It will encrypt the `rclone.conf` configuration file. This way secrets like the [](docId:OXSINcFRuVMBacPvswwNU), the encryption passphrase, and the access grant can't be stolen if an attacker gains access to your configuration file. -{% /callout %} +For additional security, you should consider using the `s) Set configuration password` option. It will encrypt the `rclone.conf` configuration file. This way secrets like the [](docId:OXSINcFRuVMBacPvswwNU), the encryption passphrase, and the access grant can't be easily stolen. ## Create a Bucket @@ -226,25 +30,25 @@ rclone mkdir waterbear:mybucket Use the `lsf` command to list all buckets. -```yaml -rclone lsf waterbear: -``` - {% callout type="info" %} Note the colon (`:`) character at the end of the command line. {% /callout %} +```bash +rclone lsf waterbear: +``` + ## Delete a Bucket Use the `rmdir` command to delete an empty bucket. -```yaml +```bash rclone rmdir waterbear:mybucket ``` Use the `purge` command to delete a non-empty bucket with all its content. -```yaml +```bash rclone purge waterbear:mybucket ``` @@ -252,7 +56,7 @@ rclone purge waterbear:mybucket Use the `copy` command to upload an object. -```yaml +```bash rclone copy --progress ~/Videos/myvideo.mp4 waterbear:mybucket/videos/ ``` @@ -262,7 +66,7 @@ The `--progress` flag is for displaying progress information. Remove it if you d Use a folder in the local path to upload all its objects. -```yaml +```bash rclone copy --progress ~/Videos/ waterbear:mybucket/videos/ ``` @@ -274,19 +78,19 @@ Only modified files will be copied. Use the `ls` command to list recursively all objects in a bucket. -```yaml +```bash rclone ls waterbear:mybucket ``` Add the folder to the remote path to list recursively all objects in this folder. -```yaml +```bash rclone ls waterbear:mybucket/videos/ ``` Use the `lsf` command to list non-recursively all objects in a bucket or a folder. -```yaml +```bash rclone lsf waterbear:mybucket/videos/ ``` @@ -294,17 +98,21 @@ rclone lsf waterbear:mybucket/videos/ Use the `copy` command to download an object. -```yaml -rclone copy --progress waterbear:mybucket/videos/myvideo.mp4 ~/Downloads/ -``` +{% callout type="info" %} +The `--progress` flag is for displaying progress information. +{% /callout %} {% callout type="info" %} -The `--progress` flag is for displaying progress information. Remove it if you don't need this information. +Using `--disable-http2` with rclone for Storj is recommended for increased transfer speeds by avoiding HTTP/2 specific issues. {% /callout %} +```bash +rclone copy --disable-http2 --progress waterbear:mybucket/videos/myvideo.mp4 ~/Downloads/ +``` + Use a folder in the remote path to download all its objects. -```yaml +```bash rclone copy --progress waterbear:mybucket/videos/ ~/Downloads/ ``` @@ -312,13 +120,13 @@ rclone copy --progress waterbear:mybucket/videos/ ~/Downloads/ Use the `deletefile` command to delete a single object. -```yaml +```bash rclone deletefile waterbear:mybucket/videos/myvideo.mp4 ``` Use the `delete` command to delete all object in a folder. -```yaml +```bash rclone delete waterbear:mybucket/videos/ ``` @@ -326,7 +134,7 @@ rclone delete waterbear:mybucket/videos/ Use the `size` command to print the total size of objects in a bucket or a folder. -```yaml +```bash rclone size waterbear:mybucket/videos/ ``` @@ -334,32 +142,28 @@ rclone size waterbear:mybucket/videos/ Use the `sync` command to sync the source to the destination, changing the destination only. Doesn’t transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary. -```yaml -rclone sync --progress ~/Videos/ waterbear:mybucket/videos/ -``` - -{% callout type="info" %} -The `--progress` flag is for displaying progress information. Remove it if you don't need this information. -{% /callout %} - -{% callout type="info" %} +{% callout type="warning" %} Since this can cause data loss, test first with the `--dry-run` flag to see exactly what would be copied and deleted. {% /callout %} +```bash +rclone sync --progress ~/Videos/ waterbear:mybucket/videos/ +``` + The sync can also be done from Storj DCS to the local file system. -```yaml +```bash rclone sync --progress waterbear:mybucket/videos/ ~/Videos/ ``` Or between two Storj DCS buckets. -```yaml +```bash rclone sync --progress waterbear-us:mybucket/videos/ waterbear-europe:mybucket/videos/ ``` Or even between another cloud storage and Storj DCS. -```yaml +```bash rclone sync --progress s3:mybucket/videos/ waterbear:mybucket/videos/ ```