Skip to content

Latest commit

 

History

History
79 lines (57 loc) · 7.28 KB

README.md

File metadata and controls

79 lines (57 loc) · 7.28 KB

Backup & Restore Procedures

Overview

This collection of procedures covers CockroachDB backup and restore operations. You will find end-to-end implementations of typical use-cases, technical considerations, caveats, and examples of SQL to launch backups and restores of CockroachDB clusters. These procedures will continue to grow as we evolve the database, integrate with other technologies, and explore trending customer use-cases that need attention.

How often should you run a backup?

This depends on the nature of your workloads, and the acceptable risk tolerance for a quantifiable volume of data-loss since your last backup. This quantifiable volume of data the maximum amount of data, measured by time of the last backup that is lost when the database was restored from the last backup-point. The elapsed time since your last backup is your RPO (recovery-point-objective) and is defined by a backup schedule where you set a time-interval. This time-interval should have a candence such that if a disaster occurs, the volume of data that is lost will have minimal or manageable impact to your business operations.

Running backup outside of the schedule

Workloads can include large batched blocks of inserts or update queries which may be transient by nature, or alternatively they can be relatively small but critical changes to your data that's tied directly to you business success. These out-of-band database updates are good candidates for manually triggered backup operations to protect these significant data events.

Understand your risk tolerance to help define a backup schedule

Regular backups are necessary to mitigate risk of data-loss even with a fault-tolerant and resilient database such as CockroachDB.

Backup procedures are part of a due-diligence process where action-plans are in place to restore data to known points in time specified by your RPO goals.

Factors to consider when defining your schedule

There is no calculator to define a schedule since many variables need to be explored, ranging from business-continuity requirements to technical considerations that may affect the backup process. The below list will help identify focus-areas and testing strategies to consider as your schedule is developed.

  • Volume/recency of transactions that need restoration due to committments through service level agreements or priority of transactions.
  • Size of data that is backed up. The more data on your platform, the longer the backup process takes and the increase in latency may be a burden to active users that are connected to the database. Note that a prior backup can be running when a new backup is triggered: Your candence must be long-enough to allow the existing backup to complete.
  • Hardware or service limitations may limit the ingress of your backup files, whether it's by physical size, network bandwidth consumed, or quota-driven cloud requests.
  • Peak vs quiet hours of operation: Automated backups are best scheduled during non-peak hours of business operations, typically outside of typical business hours to help reduce overall latency for client connections and preserve expected query peformance.
  • What to backup because CockroachDB lets you backup the entire cluster, an individual database in the cluster, or an individual table in a database.

Full and incremental cluster-wide backups

Depending on the backup frequency(eg: hourly, daily), CockroachDB (by default) assumes an incremental backup, and creates a second relative, less-frequent schedule to perform a full backup. This is specifically documented with pre-defined rules tied to the FULL BACKUP crontab parameters. While you can force a full-backup at each scheduled trigger, for most cases this is not necessary because it generates undue workload on the cluster and consumes large volumes of storage.

  • The backup process reads the entire cluster. From a byte-size perspective it can exceed GB or event TB sizes), and the backup process
  • Both full and incremental backups create independent packages and folder structures
  • Incremental backups rely on the previous full (or incremental) backup data to perform a backup from the previous point in time. Max allowed incrementals per full backup is 400, so for example if you run a backup every 5 minutes this yields 288 per-day, at which point a full-backup can be taken to reset this counter. This is listed with clarity in our Recommendations for incremental backup frequency

Backup & Restore operations using AWS S3

This set of guides define integration points between CockroachDB and AWS S3 for the data storage medium, including SQL syntax that specifically enables data flow between CockroachDB and AWS S3 storage buckets. A successful solution will require experience with AWS technologies to take advantage of enhanced security capabilities or other long-term storage requirements that you may have. Here you will find the most typical use-cases for S3 integration since it is not possible to outline every combination or scenario.

References & documentation

Cockroach Labs offers documentation into backup and restore operations that cover operational configurations, technical considerations, and usage examples. Below is a focused subset of shortcuts that are used across these runbooks.