feat(restore-from-backup): The ability for users to provide existing …

…save game data on startup added (#8)

* fix(backups): Fixed bug in IAM policy build when using s3 backups

* fix(backups): Fixed bug preventing the backup script from generating with proper inputs when using S3 backups

* feat(start-from-backup): Added the ability to use existing save files when creating the server.

README.md

@@ -25,6 +25,7 @@ This module allows you to quickly deploy an Ark Survival Ascended server on AWS.
 - Most GameUserSettings.ini settings are configurable inputs for creating a brand new configuration
 - Ability to store backups in S3 at a defined interval
 - Ability to add mods
+- Ability to start from existing save data
 
 ### Supported Maps
 | Map Name | Systemd Service Name |
@@ -34,11 +35,9 @@ This module allows you to quickly deploy an Ark Survival Ascended server on AWS.
 ### Planned Features Roadmap
 | Feature | Target Date |
 | ------- | ----------- |
-| Inputs for platform type | Jan 2024 |
 | Replace stand alone EC2 instance with Autoscaling group | Feb 2024 |
 | Parametrize Game.ini options | Feb 2024
 | Configurable Restart interval | Feb 2024 |
-| Restore from backups and use existing save games | Feb 2024 |
 | Allow users to define which map to use | Feb 2024 |
 | Allow users to launch a cluster of multiple maps | March 2024 |
 | Make compute stateless. Store data external from compute via RDS and EFS | Sometime 2024 ( I don't even know if this is possible ) |
@@ -137,6 +136,28 @@ Use the `supported_server_platforms` input to define which platforms can connect
   supported_server_platforms = ["All"]
 ```
 
+## Using Existing Save Backups and Migrating From Existing Servers
+You can use existing save data from an existing server when starting the server. This is useful if you are migrating from another hosting platform or recreating the server. The following inputs are required to do this:
+
+> [!WARNING]
+> When `backup_files_storage_type = "s3"` using The objects in the S3 bucket must not be compressed and must be in the root of the S3 bucket. The bucket's root directroy will be synced to the SaveGame directory.
+
+> [!WARNING]
+> When `backup_files_storage_type = "local"` using The objects/files in the directory you specify with `backup_files_local_path` must not be compressed. Terraform will iterate through each file in that directory and upload it to the root of an S3 bucket it creates.
+
+
+| Input | Description |
+| ------------- | ------------- |
+| `start_from_backup = "true"` | Must be set to inform Terraform that you wish to start the server with existing save data. |
+| `backup_files_storage_type = "local" or "s3"` | Valid inputs are "local" or "s3". Must be set if `start_from_backup = true`. |
+| `backup_files_local_path = "/directory/on/my/pc"` | If `backup_files_storage_type = "local"` then you must provide a path on your local host relative to the terraform working directory. |
+| `existing_backup_files_bootstrap_bucket_arn` | If `backup_files_storage_type = "s3"` then you must provide the ARN of an existing S3 bucket that contains the save game files in the root of the S3 bucket, uncompressed. |
+| `existing_backup_files_bootstrap_bucket_name` | If `backup_files_storage_type = "s3"` then you must provide the Name of an existing S3 bucket that contains the save game files in the root of the S3 bucket, uncompressed |
+
+- `backup_files_storage_type = "local"` will instruct terraform to create an S3 bucket named `ark-bootstrap-local-saves-region-accID` and upload the save files from your local PC `backup_files_local_path` directory specified to that bucket. The user_data script on the EC2 instance will download the files from that S3 bucket when the server starts and place them in the `/ark-asa/ShooterGame/Saved/SaveGames` directory.
+
+- `backup_files_storage_type = "s3"` Is informing terraform that you have an existing S3 bucket somewhere that contains the save game data. The EC2 user_data script will attempt to sync the root of that S3 bucket with the SaveGames directory of ark. That is why it is important that the objects be uncompressed and in the root of the directory. 
+
 ## Using an Existing GameUserSettings.ini
 You can use an existing GameUserSettings.ini so that the server starts with your custom settings. The following inputs are required to do this:
 

assets/1542340661.formertribeownerlog

@@ -0,0 +1 @@
+452200346,0002522bd69749ab9ffd03c48e20eca0,2024.01.21-22.22.51

docs/compute_notes.md

@@ -0,0 +1,44 @@
+# Just Some Notes
+
+## EFS?!?!
+
+## Current Compute
+- EC2
+
+Pros
+- Simple
+
+Cons
+- Not flexible
+- Updating a terraform module input that adjusts the user_data script that builds the ark configuration requires manual termination and recreation of the server.
+
+## Desired Compute
+- ASG
+
+### Driver
+
+The ark configuration is built during the first creation of the server when inputs are interpolated into the user_data template. To update the inputs users have to manually ssh and update. This creates an inconsistent state with the terraform configuration unless the user also updates the terraform configuration. Do no want to require users to update terraform code, such as adjusting Tameing speed rate, then force them to manually terminate the server and rerun the terraform apply to recreate it. 
+
+### Goal
+
+1. Users want to update an Ark setting in the .ini so they make the input change in Terraform
+2. Users terraform apply
+3. ASG Launch config/user_data template updates
+4. ASG triggers recreation of the server
+5. Recreation downloads most up-to-date save data.
+
+### Challenges
+
+- Recreating the server every time takes approx 15 - 20 minutes due to user_data downloading ark.
+- - This can be avoided by providing a custom AMI with ark preinstalled, but this requires users to initially create an AMI. Scope creep and can be potentially off putting to users.
+- No mechanism to force ark to save before destroy
+- No mechanism to start the new server with an existing save - planned feature
+
+### Pros
+- Provides flexibility. Updating a terraform module input that adjusts the user_data script that builds the ark configuration triggers an auto rebuild of the server. 
+- Immutable compute layer
+
+### Cons
+- More complex
+- Increased risk to users in form of unintended scaling up ( $$$$ )
+- Potential loss of data

docs/start_from_backup.md

@@ -0,0 +1,28 @@
+# Start From Backup
+
+## Related Ark Arguments
+
+?
+
+## Notes on Saves
+- Character profile data .arkprofil
+- Map data .ark
+- Tribe profiles .arktributetrive
+
+## Driver
+Players want to use existing save data and migrate to AWS or have lost their server, but have backups and want to restore from the backup files.
+
+## Outcome
+The ability for users to start an Ark server with existing save data
+
+## Deliverables
+- A Mechanism for users to inform Terraform on the location of existing save data
+- Supported save data locations determined and supported
+
+## Flow
+- Start server :white-check-mark:
+- Make backups :white-check-mark: ( shipped to s3)
+- Server explodes
+- Start new server
+- Place backups on new erver 
+- Start new server

examples/restore_from_backup/README.md

@@ -0,0 +1,40 @@
+# Starting an Ark Server From Existing Save Data ( Restoring from Backup )
+An example of using the inputs required to start the server with existing save data.
+
+## Details
+These inputs are required when migrating from an existing server with existing data or recovering from a deleted server.
+
+> [!WARNING]
+> When `backup_files_storage_type = "s3"` using The objects in the S3 bucket must not be compressed and must be in the root of the S3 bucket. The bucket's root directroy will be synced to the SaveGame directory.
+
+> [!WARNING]
+> When `backup_files_storage_type = "local"` using The objects/files in the directory you specify with `backup_files_local_path` must not be compressed. Terraform will iterate through each file in that directory and upload it to the root of an S3 bucket it creates.
+
+- `backup_files_storage_type = "local"` will instruct terraform to create an S3 bucket named `ark-bootstrap-local-saves-region-accID` and upload the save files from your local PC `backup_files_local_path` directory specified to that bucket. The user_data script on the EC2 instance will download the files from that S3 bucket when the server starts and place them in the `/ark-asa/ShooterGame/Saved/SaveGames` directory.
+
+- `backup_files_storage_type = "s3"` Is informing terraform that you have an existing S3 bucket somewhere that contains the save game data. The EC2 user_data script will attempt to sync the root of that S3 bucket with the SaveGames directory of ark. That is why it is important that the objects be uncompressed and in the root of the directory. 
+
+> [!WARNING]
+> When `backup_files_storage_type = "local"` using The objects/files in the directory you specify with `backup_files_local_path` must not be compressed. Terraform will iterate through each file in that directory and upload it to the root of an S3 bucket it creates.
+
+## Usage - Restore From Local Files
+Relevant inputs:
+
+```HCL
+  start_from_backup         = true
+  backup_files_storage_type = "local"
+  backup_files_local_path   = "../../assets"
+```
+
+> [!WARNING]
+> When `backup_files_storage_type = "s3"` using The objects in the S3 bucket must not be compressed and must be in the root of the S3 bucket. The bucket's root directroy will be synced to the SaveGame directory.
+
+## Usage - Restore From an Existing S3 Bucket ( Bring Your Own S3 Bucket)
+Relevant inputs:
+
+```HCL
+  start_from_backup         = true
+  backup_files_storage_type = "s3"
+  existing_backup_files_bootstrap_bucket_arn  = "arn:aws:s3:::ark-existing-s3-bucket-bootstrap"
+  existing_backup_files_bootstrap_bucket_name = "ark-existing-s3-bucket-bootstrap"
+```

examples/restore_from_backup/main.tf

@@ -0,0 +1,45 @@
+module "asa" {
+  source = "../.."
+
+  # Infrastructure inputs
+  ge_proton_version = "8-27"
+  instance_type     = "t3.xlarge"
+  create_ssh_key    = true
+  ssh_public_key    = "../../ark_public_key.pub"
+  # Ark Application inputs
+  use_battleye          = false
+  auto_save_interval    = 20.0
+  ark_session_name      = "ark-aws-ascended"
+  max_players           = "32"
+  enable_rcon           = true
+  rcon_port             = 27011
+  steam_query_port      = 27015
+  game_client_port      = 7777
+  server_admin_password = "RockwellSucks"
+  is_password_protected = true
+  join_password         = "RockWellSucks"
+  # Custom GameUserSettings.ini inputs
+  use_custom_gameusersettings        = true
+  custom_gameusersettings_s3         = true
+  game_user_settings_ini_path        = "../../TestGameUserSettings.ini"
+  custom_gameusersettings_github     = false
+  custom_gameusersettings_github_url = ""
+  # Custom Game.ini inputs
+  use_custom_game_ini       = true
+  custom_gameini_s3         = true
+  game_ini_path             = "../../TestGame.ini"
+  custom_gameini_github     = false
+  custom_gameini_github_url = ""
+  # Backup inputs
+  enable_s3_backups               = true
+  backup_s3_bucket_name           = ""
+  backup_s3_bucket_arn            = ""
+  backup_interval_cron_expression = "*/5 * * * *"
+  create_backup_s3_bucket         = true
+  s3_bucket_backup_retention      = 7
+  force_destroy                   = true
+  # Restore settings
+  start_from_backup         = true
+  backup_files_storage_type = "local"
+  backup_files_local_path   = "../../assets"
+}

examples/restore_from_backup/versions.tf

@@ -0,0 +1,3 @@
+terraform {
+  required_version = ">= 1.5.0"
+}

main.tf

@@ -53,8 +53,8 @@ module "ark_compute" {
   custom_gameini_github_url = var.custom_gameini_github_url
   # Backup inputs
   enable_s3_backups                            = var.enable_s3_backups
-  backup_s3_bucket_name                        = var.enable_s3_backups == true && var.create_backup_s3_bucket == false ? module.ark_backup.backup_s3_bucket_name[0] : var.backup_s3_bucket_name
-  backup_s3_bucket_arn                         = var.enable_s3_backups == true && var.create_backup_s3_bucket == false ? module.ark_backup.backup_s3_bucket_arn[0] : var.backup_s3_bucket_arn
+  backup_s3_bucket_name                        = var.enable_s3_backups == true && var.create_backup_s3_bucket == true ? module.ark_backup.backup_s3_bucket_name[0] : var.backup_s3_bucket_name
+  backup_s3_bucket_arn                         = var.enable_s3_backups == true && var.create_backup_s3_bucket == true ? module.ark_backup.backup_s3_bucket_arn[0] : var.backup_s3_bucket_arn
   backup_interval_cron_expression              = var.backup_interval_cron_expression
   taming_speed_multiplier                      = var.taming_speed_multiplier
   xp_multiplier                                = var.xp_multiplier
@@ -119,6 +119,11 @@ module "ark_compute" {
   structure_prevent_resource_radius_multiplier = var.structure_prevent_resource_radius_multiplier
   structure_resistance_multiplier              = var.structure_resistance_multiplier
   the_max_structure_in_range                   = var.the_max_structure_in_range
+  start_from_backup                            = var.start_from_backup
+  backup_files_storage_type                    = var.backup_files_storage_type
+  backup_files_local_path                      = var.backup_files_local_path
+  existing_backup_files_bootstrap_bucket_arn   = var.existing_backup_files_bootstrap_bucket_arn
+  existing_backup_files_bootstrap_bucket_name  = var.existing_backup_files_bootstrap_bucket_name
 }
 
 module "ark_backup" {

modules/compute/data.tf

@@ -53,9 +53,18 @@ data "template_file" "user_data_template" {
     # END Game.ini inputs
     # START backup related inputs
     enable_s3_backups               = var.enable_s3_backups
-    backup_s3_bucket_name           = var.backup_s3_bucket_name
+    backup_s3_bucket_name           = "${var.backup_s3_bucket_name}"
     backup_interval_cron_expression = var.enable_s3_backups == true ? var.backup_interval_cron_expression : ""
     # END backup related inputs
+    # START start from existing save game data
+    start_from_backup                           = "${var.start_from_backup}"
+    backup_files_storage_type                   = "${var.backup_files_storage_type}"
+    backup_files_bootstrap_bucket_arn           = var.start_from_backup == true && var.backup_files_storage_type == "local" ? "${aws_s3_bucket.ark_bootstrap[0].arn}" : "na"
+    backup_files_bootstrap_bucket_name          = var.start_from_backup == true && var.backup_files_storage_type == "local" ? "s3://${aws_s3_bucket.ark_bootstrap[0].bucket}" : "na"
+    backup_files_local_path                     = var.start_from_backup == true && var.backup_files_storage_type == "local" ? "${var.backup_files_local_path}" : "na"
+    existing_backup_files_bootstrap_bucket_arn  = var.start_from_backup == true && var.backup_files_storage_type == "s3" ? "${var.existing_backup_files_bootstrap_bucket_arn}" : "na"
+    existing_backup_files_bootstrap_bucket_name = var.start_from_backup == true && var.backup_files_storage_type == "s3" ? "s3://${var.existing_backup_files_bootstrap_bucket_name}" : "na"
+    # END start from existing save game data
     taming_speed_multiplier                      = "${var.taming_speed_multiplier}"
     xp_multiplier                                = "${var.xp_multiplier}"
     server_pve                                   = "${var.server_pve}"

modules/compute/iam.tf

@@ -1,13 +1,13 @@
 resource "aws_iam_role" "instance_role" {
-  count = var.custom_gameusersettings_s3 == true || var.custom_gameini_s3 == true ? 1 : 0
+  count = var.custom_gameusersettings_s3 == true || var.custom_gameini_s3 == true || var.start_from_backup == true || var.enable_s3_backups == true ? 1 : 0
 
   name               = "ark-instance-role-${data.aws_region.current.name}"
   path               = "/"
   assume_role_policy = file("${path.module}/templates/ark-instance-role.json")
 }
 
 resource "aws_iam_role_policy" "instance_role_policy" {
-  count = var.use_custom_gameusersettings == true && var.custom_gameusersettings_s3 == true || var.use_custom_gameusersettings == true && var.custom_gameini_s3 == true ? 1 : 0
+  count = var.use_custom_gameusersettings == true && var.custom_gameusersettings_s3 == true || var.use_custom_gameusersettings == true && var.custom_gameini_s3 == true || var.start_from_backup == true || var.enable_s3_backups == true ? 1 : 0
 
   name   = "ark-instance-role-policy-${data.aws_region.current.name}"
   policy = data.aws_iam_policy_document.ark_policy[0].json
@@ -29,15 +29,15 @@ resource "aws_iam_role_policy" "instance_role_policy" {
 
 
 resource "aws_iam_instance_profile" "instance_profile" {
-  count = var.use_custom_gameusersettings == true && var.custom_gameusersettings_s3 == true || var.use_custom_gameusersettings == true && var.custom_gameini_s3 == true ? 1 : 0
+  count = var.use_custom_gameusersettings == true && var.custom_gameusersettings_s3 == true || var.use_custom_gameusersettings == true && var.custom_gameini_s3 == true || var.start_from_backup == true || var.enable_s3_backups == true ? 1 : 0
 
   name = "ark-instance-profile-${data.aws_region.current.name}"
   path = "/"
   role = aws_iam_role.instance_role[0].name
 }
 
 data "aws_iam_policy_document" "ark_policy" {
-  count = var.use_custom_gameusersettings == true && var.custom_gameusersettings_s3 == true || var.use_custom_gameusersettings == true && var.custom_gameini_s3 == true ? 1 : 0
+  count = var.use_custom_gameusersettings == true && var.custom_gameusersettings_s3 == true || var.use_custom_gameusersettings == true && var.custom_gameini_s3 || var.start_from_backup == true || var.enable_s3_backups == true == true ? 1 : 0
   statement {
     sid = "InteractWithS3"
 
@@ -56,6 +56,10 @@ data "aws_iam_policy_document" "ark_policy" {
       var.custom_gameini_s3 == true ? "${aws_s3_bucket.ark[0].arn}/*" : "",
       var.enable_s3_backups == true ? var.backup_s3_bucket_arn : "",
       var.enable_s3_backups == true ? "${var.backup_s3_bucket_arn}/*" : "",
+      var.start_from_backup == true && var.backup_files_storage_type == "local" ? aws_s3_bucket.ark_bootstrap[0].arn : "",
+      var.start_from_backup == true && var.backup_files_storage_type == "local" ? "${aws_s3_bucket.ark_bootstrap[0].arn}/*" : "",
+      var.start_from_backup == true && var.backup_files_storage_type == "s3" ? var.existing_backup_files_bootstrap_bucket_arn : "",
+      var.start_from_backup == true && var.backup_files_storage_type == "s3" ? "${var.existing_backup_files_bootstrap_bucket_arn}/*" : "",
     ])
   }
 }

modules/compute/s3.tf

@@ -1,12 +1,12 @@
 resource "aws_s3_bucket" "ark" {
-  count = var.use_custom_gameusersettings == true && var.custom_gameusersettings_s3 == true || var.use_custom_gameusersettings == true && var.custom_gameini_s3 == true ? 1 : 0
+  count = var.use_custom_gameusersettings == true && var.custom_gameusersettings_s3 == true || var.use_custom_game_ini == true && var.custom_gameini_s3 == true ? 1 : 0
 
   bucket = "ark-app-${data.aws_region.current.name}-${data.aws_caller_identity.current.account_id}"
 
 }
 
 resource "aws_s3_bucket_versioning" "ark" {
-  count = var.use_custom_gameusersettings == true && var.custom_gameusersettings_s3 == true || var.use_custom_gameusersettings == true && var.custom_gameini_s3 == true ? 1 : 0
+  count = var.use_custom_gameusersettings == true && var.custom_gameusersettings_s3 == true || var.use_custom_game_ini == true && var.custom_gameini_s3 == true ? 1 : 0
 
   bucket = aws_s3_bucket.ark[0].id
 
@@ -24,9 +24,40 @@ resource "aws_s3_object" "gameusersettings" {
 }
 
 resource "aws_s3_object" "gameini" {
-  count = var.use_custom_gameusersettings == true && var.custom_gameini_s3 == true ? 1 : 0
+  count = var.use_custom_game_ini == true && var.custom_gameini_s3 == true ? 1 : 0
 
   bucket = aws_s3_bucket.ark[0].id
   key    = "Game.ini"
   source = var.game_ini_path
 }
+
+
+### Start From Backup ###
+resource "aws_s3_bucket" "ark_bootstrap" {
+  count = var.start_from_backup == true && var.backup_files_storage_type == "local" ? 1 : 0
+
+  bucket = "ark-bootstrap-local-saves-${data.aws_region.current.name}-${data.aws_caller_identity.current.account_id}"
+
+}
+
+resource "aws_s3_bucket_versioning" "ark_bootstrap" {
+  count = var.start_from_backup == true && var.backup_files_storage_type == "local" ? 1 : 0
+
+  bucket = aws_s3_bucket.ark_bootstrap[0].id
+
+  versioning_configuration {
+    status = "Disabled"
+  }
+}
+
+locals {
+  files = var.start_from_backup == true && var.backup_files_storage_type == "local" ? fileset(var.backup_files_local_path, "*") : []
+}
+
+resource "aws_s3_object" "bootstrap_savegame_files" {
+  for_each = { for f in local.files : f => f }
+
+  bucket = aws_s3_bucket.ark_bootstrap[0].id
+  key    = basename(each.value)
+  source = "${var.backup_files_local_path}/${each.value}"
+}