diff --git a/docs/platform/concepts/byoc.md b/docs/platform/concepts/byoc.md index 2d18e7167..5eccebb41 100644 --- a/docs/platform/concepts/byoc.md +++ b/docs/platform/concepts/byoc.md @@ -141,6 +141,11 @@ to work properly (supporting HA signaling to the Aiven management node and RPM d from Aiven repositories). ::: +Object storage in your AWS cloud account is where service's +[backups](/docs/platform/concepts/byoc#byoc-service-backups) and +[cold data](/docs/platform/howto/byoc/store-data#byoc-tiered-storage) are stored using +two S3 buckets. + @@ -154,6 +159,12 @@ through the public internet: the Aiven control plane connects to the nodes using the public address, and the Aiven management plane can access the service VMs directly. To restrict access to your service, you can use the [IP filter](/docs/platform/howto/restrict-access). + +Object storage in your AWS cloud account is where service's +[backups](/docs/platform/concepts/byoc#byoc-service-backups) and +[cold data](/docs/platform/howto/byoc/store-data#byoc-tiered-storage) are stored using +two S3 buckets. + @@ -183,6 +194,11 @@ to work properly (supporting HA signaling to the Aiven management node and RPM d from Aiven repositories). ::: +Object storage in your Google Cloud organization is +where service's [backups](/docs/platform/concepts/byoc#byoc-service-backups) and +[cold data](/docs/platform/howto/byoc/store-data#byoc-tiered-storage) are stored using +Google **Cloud Storage** buckets. + @@ -196,6 +212,12 @@ through the public internet: the Aiven control plane connects to the nodes using the public address, and the Aiven management plane can access the service VMs directly. To restrict access to your service, you can use the [IP filter](/docs/platform/howto/restrict-access). + +Object storage in your Google Cloud organization is +where service's [backups](/docs/platform/concepts/byoc#byoc-service-backups) and +[cold data](/docs/platform/howto/byoc/store-data#byoc-tiered-storage) are stored using +Google **Cloud Storage** buckets. + @@ -208,10 +230,12 @@ All Aiven communication is encrypted. Depending on the BYOC service, Aiven takes [regular service backups](/docs/platform/concepts/service_backups) to enable forking, point in time recovery (PITR), and disaster recovery. -These backups by default do not reside in your cloud. If there is a -requirement to have all backups in your own cloud account, it's still possible. -To accomplish this, Aiven needs read-write permissions to access the object storage on -your cloud account. + +BYOC-hosted services have user-owned backups stored in object storage in your AWS +account or your Google Cloud organization. Backups reside in: + +- S3 buckets for AWS BYOC environments +- Cloud Storage buckets for Google Cloud BYOC environments :::important @@ -232,4 +256,4 @@ Aiven deployment model. - [Bring your own cloud networking and security](/docs/platform/howto/byoc/networking-security) - [Enable bring your own cloud (BYOC)](/docs/platform/howto/byoc/enable-byoc) - [Create a custom cloud in Aiven](/docs/platform/howto/byoc/create-cloud/create-custom-cloud) -- [Manage services hosted in custom clouds](/docs/platform/howto/byoc/manage-byoc-service) +- [Store BYOC data in your own cloud account](/docs/platform/howto/byoc/store-data) diff --git a/docs/platform/concepts/service_backups.md b/docs/platform/concepts/service_backups.md index 450e95693..9b8040dc5 100644 --- a/docs/platform/concepts/service_backups.md +++ b/docs/platform/concepts/service_backups.md @@ -334,3 +334,9 @@ backups, see For more information on Aiven for ClickHouse backups, see [Backup and restore](/docs/products/clickhouse/concepts/disaster-recovery). + +## BYOC service backups + +Learn about +[backups for services hosted in custom clouds](/docs/platform/concepts/byoc#byoc-service-backups) +or [bring your own cloud (BYOC)](/docs/platform/concepts/byoc) environments. diff --git a/docs/platform/howto/byoc/create-cloud/create-aws-custom-cloud.md b/docs/platform/howto/byoc/create-cloud/create-aws-custom-cloud.md index 99da0987d..1d9848813 100644 --- a/docs/platform/howto/byoc/create-cloud/create-aws-custom-cloud.md +++ b/docs/platform/howto/byoc/create-cloud/create-aws-custom-cloud.md @@ -478,6 +478,20 @@ In the **Create custom cloud** wizard: cannot change the BYOC VPC CIDR block after your custom cloud is created. + - Remote storage (BYOC-hosted) + + By default, the following data is stored in the BYOC object storage in your own + cloud account: + + - [Cold data](/docs/platform/howto/byoc/store-data) + - [Service backups](/docs/platform/concepts/byoc#byoc-service-backups) + + :::note + - Data is stored in your BYOC object storage using one S3 bucket per custom cloud. + - Permissions for S3 bucket management will be included in the Terraform + infrastructure template to be generated upon completing this step. + ::: + Click **Generate template**. Your IaC Terraform template gets generated based on your inputs. You can diff --git a/docs/platform/howto/byoc/create-cloud/create-google-custom-cloud.md b/docs/platform/howto/byoc/create-cloud/create-google-custom-cloud.md index 5ddc281d6..2afa668e9 100644 --- a/docs/platform/howto/byoc/create-cloud/create-google-custom-cloud.md +++ b/docs/platform/howto/byoc/create-cloud/create-google-custom-cloud.md @@ -158,6 +158,21 @@ In the **Create custom cloud** wizard: cannot change the BYOC VPC CIDR block after your custom cloud is created. + - Remote storage (BYOC-hosted) + + By default, the following data is stored in the BYOC object storage in your own + cloud account: + + - [Cold data](/docs/platform/howto/byoc/store-data) + - [Service backups](/docs/platform/concepts/byoc#byoc-service-backups) + + :::note + - Data is stored in your BYOC object storage using one Cloud Storage bucket per + custom cloud. + - Permissions for Cloud Storage bucket management will be included in the Terraform + infrastructure template to be generated upon completing this step. + ::: + Click **Generate template**. Your infrastructure Terraform template gets generated based on your inputs. You can diff --git a/docs/platform/howto/byoc/store-data.md b/docs/platform/howto/byoc/store-data.md new file mode 100644 index 000000000..526554da8 --- /dev/null +++ b/docs/platform/howto/byoc/store-data.md @@ -0,0 +1,73 @@ +--- +title: Store BYOC data in your own cloud account +sidebar_label: BYOC tiered storage +keywords: [bring your own cloud, byoc, custom cloud, BYOC cloud, object storage, tiered storage, bucket] +--- + +import ConsoleLabel from "@site/src/components/non-swizzled/ConsoleIcons"; + +BYOC environments use the tiered storage capability for data allocation. Cold data in your +custom cloud is stored in your AWS cloud account or your Google Cloud organization. + +## BYOC tiered storage + +:::important +[BYOC](/docs/platform/concepts/byoc) tiered storage is only supported for +[Aiven for Apache Kafka](/docs/products/kafka/howto/kafka-tiered-storage-get-started) and +[Aiven for ClickHouse](/docs/products/clickhouse/concepts/clickhouse-tiered-storage). +::: + +To store data, [BYOC](/docs/platform/concepts/byoc) environments use tiered storage, a +data allocation mechanism for improved efficiency and cost optimization of data management. +When enabled, tiered storage allows moving data automatically between hot storage (for +frequently accessed, critical, and often updated data) and cold storage (for rarely +accessed, static, or archived data). + +Cold data of BYOC-hosted services is stored in object storage in your AWS cloud +account or your Google Cloud organization. One bucket is created per custom cloud. + +:::note + +- Tiered storage enabled on non-BYOC services is owned by Aiven and as such doesn't allow + to store cold data in your own cloud account. +- Non-BYOC services with Aiven-owned tiered storage cannot be migrated to BYOC. + +::: + +To use tiered storage in an BYOC-hosted service, tiered storage needs to be enabled both +[in your custom cloud](/docs/platform/howto/byoc/store-data#enable-tiered-storage-in-a-custom-cloud) +and +[in the BYOC-hosted service](/docs/platform/howto/byoc/store-data#enable-tiered-storage-on-a-service). + +## Enable tiered storage in a custom cloud + +- **New custom clouds**: Tiered storage is enabled by default in all new custom + clouds so you can proceed to + [enabling tiered storage on a service](/docs/platform/howto/byoc/store-data#enable-tiered-storage-on-a-service). +- **Existing custom clouds with no tiered storage support**: + [Contact the Aiven support team](mailto:support@aiven.io) to request enabling tiered + storage in your custom cloud. + +## Enable tiered storage on a service + +### Prerequisites + +- At least one [custom cloud](/docs/platform/howto/byoc/create-cloud/create-custom-cloud) +- At least one [Aiven-manged service](/docs/platform/howto/create_new_service), either + Aiven for Apache Kafka® or Aiven for ClickHouse®, hosted in an custom cloud + + :::note + If your Aiven-managed service is not hosted in a custom cloud, you can + [migrate it](/docs/platform/howto/byoc/manage-byoc-service#migrate-an-existing-service-to-a-custom-cloud). + ::: + +### Activate tiered storage + +- [Enable for Aiven for Apache Kafka](/docs/products/kafka/howto/enable-kafka-tiered-storage) +- [Enable for Aiven for Clickhouse](/docs/products/clickhouse/howto/enable-tiered-storage) + +## Related pages + +- [Bring your own cloud networking and security](/docs/platform/howto/byoc/networking-security) +- [View the status of a custom cloud](/docs/platform/howto/byoc/view-custom-cloud-status) +- [Manage services hosted in custom clouds](/docs/platform/howto/byoc/manage-byoc-service) diff --git a/sidebars.ts b/sidebars.ts index 1cd800dc5..256a404df 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -308,6 +308,7 @@ const sidebars: SidebarsConfig = { 'platform/howto/byoc/assign-project-custom-cloud', 'platform/howto/byoc/add-customer-info-custom-cloud', 'platform/howto/byoc/tag-custom-cloud-resources', + 'platform/howto/byoc/store-data', 'platform/howto/byoc/rename-custom-cloud', 'platform/howto/byoc/download-infrastructure-template', 'platform/howto/byoc/delete-custom-cloud', diff --git a/static/images/content/figma/byoc-aws-private.png b/static/images/content/figma/byoc-aws-private.png index 743eca95f..30f096c16 100644 Binary files a/static/images/content/figma/byoc-aws-private.png and b/static/images/content/figma/byoc-aws-private.png differ diff --git a/static/images/content/figma/byoc-aws-public.png b/static/images/content/figma/byoc-aws-public.png index 3f8453b0a..4e2c6950c 100644 Binary files a/static/images/content/figma/byoc-aws-public.png and b/static/images/content/figma/byoc-aws-public.png differ diff --git a/static/images/content/figma/byoc-gcp-private.png b/static/images/content/figma/byoc-gcp-private.png index 304c31fb1..b26040329 100644 Binary files a/static/images/content/figma/byoc-gcp-private.png and b/static/images/content/figma/byoc-gcp-private.png differ diff --git a/static/images/content/figma/byoc-gcp-public.png b/static/images/content/figma/byoc-gcp-public.png index 6f2cf2d6a..53c4f7e9e 100644 Binary files a/static/images/content/figma/byoc-gcp-public.png and b/static/images/content/figma/byoc-gcp-public.png differ