APT and YUM repo backups

[FooBarWidget]

We should have a system for regularly backing up the APT and YUM repos.

Use case

Backups are to protect us against the following scenarios:

1. Somebody accidentally deletes all data in the APT and YUM buckets.
2. An attacker gains access to one of the Infra Maintainers' credentials, and replaces packages with compromised versions or destroys data.

Backup schedule

Given our release speed, I think that backing up once a week is reasonable.

Security

For security reasons, neither any systems in the fullstaq-ruby Google Cloud project, nor any Infra Maintainers, must be able to delete or overwrite old backups.

Encryption is not necessary, since it's public data.

Documentation

There must also be documentation on:

• How the backup system works.
• How to perform a restore.

Implementation

We periodically backup un-backed-up versions in the APT and YUM repo, into other buckets that's specifically for storing backups. These other buckets:

• Have colder storage classes.
• Are not writable by the "Server Edition CI Bot" and the "Infrastructure CI bot" service accounts. They are only writable by Infra Owners.

A script that periodically performs gsutil rsync should be enough.

We can execute this script from a special Github repo's Github Actions. This repo is only writable by Infra Owners.