This is "molibackup" also known as "modular libre backup". It is a golang based backup utility, hence it should be able to run on all major paltforms and architectures.
In the current version, this program provides features for creating and managing snapshots of EBS volumes in AWS (Amazon Web Services). It is able to find all volumes attached to either one specific instance or all instances which have particular tags, create a snapshot of each volume, and then delete snapshots of these volumes after a retention period.
The documentation for this program, and the configuration details in particular are subject to change over time. Please make sure you are reading the version of the documentation that corresponds to the version of the program you are using. You can find the documentation for any version of this program by accessing tagged versions of the documentation files in the git repository: https://github.com/fdupoux/molibackup
This program is stateless, it does not require any database or state file to track the state of its backups. You just need the static binary, a yaml configuration file, and access to the required service APIs.
Installing this program involves the following steps:
- Downloading and copying the static binary to your system
- Creating a yaml configuration file on your system
- Creating a cron job (or similar) to run it on a regular basis
- Providing access to the service APIs with sufficient credentials
As this is a golang program, the binaries are static with no runtime dependencies.
Hence the application can be installed by just copying the binary to a location such
as /usr/local/sbin.
This application is using a yaml configuration file. By default it will try to find the configuration file in the following locations:
- /etc/molibackup/molibackup.yaml
- ${HOME}/molibackup/molibackup.yaml
- directory where the binary is located
You can also specify the path to the configuration file on the command line:
$ /usr/local/sbin/molibackup -c /etc/molibackup/molibackup.yaml
The configuration file is made of a global section as well as a jobs section
which defines one or multiple backup jobs. Here is an example:
# cat /etc/molibackup/molibackup.yaml
---
global:
loglevel: info
jobs:
myjob01:
module: ebs-snapshot
enabled: true
dryrun: false
retention: 90
aws_region: "us-west-2"
accesskey_id: "MyAccessKeyId"
accesskey_secret: "MyAccessKeySecret"
instance_id: "i-01233456789abcdef"
myjob02:
module: ebs-snapshot
enabled: true
dryrun: false
retention: 90
aws_region: "us-west-2"
accesskey_id: "MyAccessKeyId"
accesskey_secret: "MyAccessKeySecret"
instance_id: "local"
Each job comes with several options which are either mandatory or optional. The
module option is mandatory and it tells the program what type of backup to
create. At this stage, this program comes with only one module: ebs-snapshot.
The enabled and dryrun ones are optional, and their respective default values
are true and false. They allow you to disable a backup job, and to do a dry run
to see what the program would do without actually do anything. The other options in
the job confguration are specific to each type of backup job, and these are documented
in the sections corresponding to each type of backup.
You need to create a cron job (or use any alternative scheduler) to execute the program automatically. Here is an example of a cronjob which runs the program daily at 4am:
# cat /etc/cron.d/molibackup
0 4 * * * root /usr/local/sbin/molibackup -c /etc/molibackup/molibackup.yaml >> /var/log/molibackup.log 2>&1
The example above configures the cron job to run as the root user as this should
exist on all linux systems. But the program does not need root privileges to run.
Hence it is recommended to run it as an ordinary user. Just make sure this ordinary
user has sufficient permissions to read the configuration file and write to the log
file.
If the backup module you are using requires access to some service APIs, make sure these conditions are satisfied. Please refer to the module specific documentation below for more details.
This program returns the following exit status depending on the success or failure:
- 0 => The program has executed all backup jobs without any error
- 1 => Failed to read the configuration file (file not found or invalid configuration)
- 2 => The program has attempted to run the backup jobs but there were some errors
Static binaries for this program are provided on the official project page for the most popular platforms and architectures so you do not have to build it yourself. But you can compile this program from sources if there is no official binary for your platform or architecture, or if you want to build it for any other reason.
You need golang version 1.19 or more recent, as well as make and sed in order to
build this program from sources. These dependencies must either be present on your build
system, or you can use a docker image which comes with all these dependencies.
You can build a binary for your local platform by running make build if you have all
the build dependencies installed on your system. Or you can use make docker-build if
you want to build the program using the recommended docker image.
It is possible to create reproducible builds. It means the compiled binaries should always be strictly identical if you compile this program from the same sources multiple times. This allows you to check that the binary you are using was built from the official sources without any malicious modification. For this to work you have to use the same version of golang with the same sources and with the correct compilation options. To rebuild an official release in such a way, you have to get the correct sources by using either the sources from an official release source file, or by getting the sources from the tag corresponding to the release in the git repository. You then have to follow the compilation instructions included in the sources for the version you want to rebuild.
Here is how to create reproducible builds for the current version. You should run
make docker-release to rebuild the binaries with the same build command and the same
docker image that was used to produce the official release. You can then use the
sha256sum command to make sure the binaries you produced match the checksums of the
official binary files.
This program comes with a module named ebs-snapshot which is able to create and rotate
snapshots of EBS volumes in AWS. It finds one or multiple EC2 instances based on the
criteria specified in the configuration, then it finds all EBS volumes attached to these
EC2 instances and it creates a snapshots of some or all of these EBS volumes depending on
the configuration. The snapshots are named after the name of the corresponding volumes
with the date of the backup at the end. Finally it deletes snapshots which are older than
the retention period.
Here is an example of a configuration file for running multiple jobs that create EBS snapshots:
# cat /etc/molibackup/molibackup.yaml
---
global:
loglevel: info
jobs:
myjob01:
module: ebs-snapshot
enabled: true
dryrun: false
retention: 180
aws_region: "us-west-2"
accesskey_id: "MyAccessKeyId"
accesskey_secret: "MyAccessKeySecret"
instance_id: "i-01233456789abcdef"
myjob02:
module: ebs-snapshot
enabled: true
dryrun: false
retention: 60
aws_region: "us-west-2"
instance_id: "local"
myjob03:
module: ebs-snapshot
enabled: true
dryrun: false
retention: 90
aws_region: "us-west-2"
accesskey_id: "MyAccessKeyId"
accesskey_secret: "MyAccessKeySecret"
instance_tags:
Molibackup_enabled: "true"
Environment: "production"
volume_tags:
Molibackup_enabled: "true"
lock_mode: "governance"
lock_duration: 7
The aws_region attribute is mandatory. The AWS Access Key pair details are required
unless you run the program on an EC2 instance which is attached to an IAM role which
has sufficient privileges to perform all the actions.
The instance_id, instance_tags and volume_tags attributes are optional. They are used
to restrict the scope of the job. For example you can specify one or multiple tags using
instance_tags so the program resctrits the backup to instances for which all tags specified
match. The same principle applies to volume_tags which allows you to select which volumes
must be included in the scope of the job. The tags are case sensitive so please make sure
the values in the configuration matches the case of the actual tags you have created on
your instances and volumes.
The retention option speficies the retention period expressed in days. For example if
you set retention: 90 it will delete snapshots which were created more than 90 days ago.
If you do not specify the retention attribute, it will use 30 days as the default value.
The lock_mode and lock_duration attributes are optional. They allow you to lock an EBS
snapshot for a duration express in days in order to prevent accidental or malicious deletion
of snapshots during this period. You should set lock_mode to either governance or
compliance if you want to lock your snapshots.
You can either install this program to run on each EC2 instances that needs to be backed up, or you can install it on an server that will be responsible for creating the backups for multiple EC2 instances in a particular region of your AWS account.
If you want this program to create snapshots of the volumes of a single EC2 instance, you
should use the instance_id attribute in the job configuration to target this single
instance. You should either set the instance_id attribute to a specific instance ID
or you can use the special keyword local so the program uses the instance meta-data to
automatically detects the ID of the instance where it is running.
Alternatively you can configure this program to create backups of multiple EC2 instances.
In that case you should use the instance_tags attribute instead of instance_id. You will
have to create tags on all EC2 instance that needs to be backed up so the program can
determine which instances must be included in the scope of the backup job. By default the
program will create snapshots of all EBS volumes which are attached to these EC2 instances.
If you want to control which EBS volumes must be included in the backup job, you should
create tags on all EBS volumes that you want to be backed up and use the volume_tags
attribute in the job configuration to tell the program which volumes must be included.
Please refer to the configuration section above for specific examples.
The program creates and rotates snapshots of EBS volumes which meet the conditions specified in the jobs configurations.
First it finds all EC2 instances that meet the conditions configured using instance_id
and/or instance_tags. Then it finds all EBS volumes which are attached to these EC2
instances and which meet the volume tags specified with volume_tags if this option is
present.
After it has found a list of all volumes that are included in the scope of the backup job, it creates one new snapshot of each volume. Then it finds all snapshots that have already been created for these volumes, and it deletes snapshots which are older than the retention period.
The ebs-snapshot module uses the AWS APIs to create an manage snapshots of EBS Volumes.
Hence it requires an IAM Role with sufficient AWS credentials to perform these actions.
These credentials can be provided either explicitly through an AWS Access Key pair, or
implicitly by running the program on an EC2 instance which has the IAM Role attached
to it via an instance profile. The IAM Role you are using requires the following permissions
so the program is able to run successfully:
ec2:CreateSnapshot
ec2:LockSnapshot
ec2:CreateTags
ec2:DeleteSnapshot
ec2:DescribeInstances
ec2:DescribeSnapshotAttribute
ec2:DescribeSnapshots
ec2:DescribeSnapshotsAttribute
ec2:DescribeVolumeAttribute
ec2:DescribeVolumeStatus
ec2:DescribeVolumes
ec2:DescribeTags
ec2:ResetSnapshotAttribute
Here is an example of what this backup module does when it is configured to manage snapshots of three EBS volumes with a retention of five days. It first creates a new snapshot of each volume, then check how old each existing snapshot for these volumes are, and delete these which are more than five days old:
[2024/01/21T02:00:01.416] [INFO] molibackup version 0.1.0 built with go1.20.13 starting ...
[2024/01/21T02:00:01.420] [INFO] Found configuration file in /etc/molibackup/molibackup-test.yaml
[2024/01/21T02:00:01.422] [INFO] Running job "job01_websrv" ...
[2024/01/21T02:00:04.651] [INFO] Successfully created snapshot "snap-0018972b533274049" of volume "vol-03774e949840089cb"
[2024/01/21T02:00:05.246] [INFO] Successfully created snapshot "snap-0ff55e42f3d90ac21" of volume "vol-02efc43a09fff36eb"
[2024/01/21T02:00:05.789] [INFO] Successfully created snapshot "snap-0e00e1692d53abf05" of volume "vol-00215de836256ae7b"
[2024/01/21T02:00:07.303] [INFO] Deleted snapshot: id="snap-01376e71c9d7c6263" desc="zl-websrv-t01-root-2024-01-15T02:00:05Z" age=6 retention=5
[2024/01/21T02:00:07.303] [INFO] Keeping snapshot: id="snap-000b60f2b6bef27b8" desc="zl-websrv-t01-root-2024-01-16T02:00:04Z" age=5 retention=5
[2024/01/21T02:00:07.303] [INFO] Keeping snapshot: id="snap-039eb38549ee363a1" desc="zl-websrv-t01-root-2024-01-17T02:00:05Z" age=4 retention=5
[2024/01/21T02:00:07.303] [INFO] Keeping snapshot: id="snap-055d813b7b4e5429c" desc="zl-websrv-t01-root-2024-01-18T02:00:05Z" age=3 retention=5
[2024/01/21T02:00:07.303] [INFO] Keeping snapshot: id="snap-062e5b24ff9ea53e3" desc="zl-websrv-t01-root-2024-01-19T02:00:06Z" age=2 retention=5
[2024/01/21T02:00:07.303] [INFO] Keeping snapshot: id="snap-010d48fa1e4029bce" desc="zl-websrv-t01-root-2024-01-20T02:00:05Z" age=1 retention=5
[2024/01/21T02:00:07.303] [INFO] Keeping snapshot: id="snap-0e00e1692d53abf05" desc="zl-websrv-t01-root-2024-01-21T02:00:05Z" age=0 retention=5
[2024/01/21T02:00:07.766] [INFO] Deleted snapshot: id="snap-04adb311055e76c9d" desc="zl-websrv-t02-data-2024-01-15T02:00:04Z" age=6 retention=5
[2024/01/21T02:00:07.766] [INFO] Keeping snapshot: id="snap-0f5dec4ee66d7bd3f" desc="zl-websrv-t02-data-2024-01-16T02:00:04Z" age=5 retention=5
[2024/01/21T02:00:07.766] [INFO] Keeping snapshot: id="snap-0a2b5973086091a2d" desc="zl-websrv-t02-data-2024-01-17T02:00:04Z" age=4 retention=5
[2024/01/21T02:00:07.766] [INFO] Keeping snapshot: id="snap-092da8f934d4bddd1" desc="zl-websrv-t02-data-2024-01-18T02:00:04Z" age=3 retention=5
[2024/01/21T02:00:07.766] [INFO] Keeping snapshot: id="snap-0435d629492b909a0" desc="zl-websrv-t02-data-2024-01-19T02:00:05Z" age=2 retention=5
[2024/01/21T02:00:07.766] [INFO] Keeping snapshot: id="snap-0ece3d4d950c2f619" desc="zl-websrv-t02-data-2024-01-20T02:00:05Z" age=1 retention=5
[2024/01/21T02:00:07.766] [INFO] Keeping snapshot: id="snap-0ff55e42f3d90ac21" desc="zl-websrv-t02-data-2024-01-21T02:00:04Z" age=0 retention=5
[2024/01/21T02:00:08.202] [INFO] Deleted snapshot: id="snap-092f130a665de656a" desc="zl-websrv-t02-root-2024-01-15T02:00:04Z" age=6 retention=5
[2024/01/21T02:00:08.202] [INFO] Keeping snapshot: id="snap-0a2728fefbb2d5495" desc="zl-websrv-t02-root-2024-01-16T02:00:03Z" age=5 retention=5
[2024/01/21T02:00:08.202] [INFO] Keeping snapshot: id="snap-0c450b683af663fa4" desc="zl-websrv-t02-root-2024-01-17T02:00:04Z" age=4 retention=5
[2024/01/21T02:00:08.202] [INFO] Keeping snapshot: id="snap-046e47186af834f09" desc="zl-websrv-t02-root-2024-01-18T02:00:03Z" age=3 retention=5
[2024/01/21T02:00:08.202] [INFO] Keeping snapshot: id="snap-0fd788321738b9ca3" desc="zl-websrv-t02-root-2024-01-19T02:00:05Z" age=2 retention=5
[2024/01/21T02:00:08.202] [INFO] Keeping snapshot: id="snap-0d0438e486b97f5f6" desc="zl-websrv-t02-root-2024-01-20T02:00:04Z" age=1 retention=5
[2024/01/21T02:00:08.202] [INFO] Keeping snapshot: id="snap-0018972b533274049" desc="zl-websrv-t02-root-2024-01-21T02:00:04Z" age=0 retention=5
[2024/01/21T02:00:08.202] [INFO] Have successfully executed 1 jobs