ENG-893 AWS Backup Restore Testing Validation & Integrity Design

docs/manual-restore-validation.md

@@ -0,0 +1,137 @@
+# Manual Restore Validation Design
+
+## 1. Purpose
+
+Provide a light‑weight, on-demand restore + validation workflow where the **customer supplies their own validation Lambda**. This complements automated restore testing plans by enabling ad-hoc integrity checks (e.g. regression assessment after schema change, pre-cutover rehearsal) without standing orchestration state machines.
+
+## 2. Overview
+
+Flow (single resource type per invocation):
+
+1. Operator (or CI job) invokes Orchestrator Lambda with optional `recoveryPointArn`.
+2. Orchestrator chooses recovery point (latest if unspecified) from a backup vault.
+3. Starts restore job using AWS Backup `StartRestoreJob`.
+4. Polls status (`DescribeRestoreJob`) until terminal state.
+5. Invokes customer validator Lambda with contextual payload.
+6. Normalises validator response -> calls `PutRestoreValidationResult`.
+7. Returns composite result to caller (for CLI / API inspection).
+
+No Step Functions required for typical short restore + validation cycles; for long running (>15 min) scenarios Step Functions could replace polling.
+
+## 3. Roles & Responsibilities
+
+| Component | Responsibility |
+|-----------|----------------|
+| Orchestrator Lambda | Restore initiation, polling, validator invocation, publishing result |
+| Customer Validator Lambda | Domain/resource-specific integrity checks (S3 object presence, record counts, hashes, etc.) |
+| AWS Backup | Recovery point catalog & restore execution |
+| IAM | Enforces least privilege for restore & validation actions |
+
+## 4. Invocation Payload (Optional Fields)
+
+```json
+{
+  "recoveryPointArn": "arn:aws:backup:...:recovery-point:...",  // optional override
+  "expectedKeys": ["path/example1.txt", "path/example2.txt"],   // validator-specific
+  "expectedMinObjects": 10                                       // optional fallback
+}
+```
+
+## 5. Validator Contract
+
+Input delivered to customer Lambda (superset of invocation + restore context):
+
+```json
+{
+  "restoreJobId": "...",
+  "recoveryPointArn": "...",
+  "resourceType": "S3",
+  "createdResourceArn": "arn:aws:s3:::restored-bucket",
+  "targetBucket": "restored-bucket",
+  "s3": { "bucket": "restored-bucket" },
+  "expectedKeys": ["..."],
+  "expectedMinObjects": 10
+}
+```
+
+Return:
+
+```json
+{ "status": "SUCCESSFUL|FAILED|SKIPPED", "message": "summary", "details": { } }
+```
+Status mapping is case-insensitive; unknown maps to FAILED.
+
+## 6. Security Considerations
+
+- Orchestrator policy limited to listing recovery points, starting & describing restore jobs, publishing validation, invoking a single validator ARN.
+- Validator policy scoped to specific target bucket ARNs (S3 example).
+- Sensitive data avoidance: orchestrator does not log object contents, only metadata.
+- Optionally use a dedicated IAM restore role if restore requires cross-service access.
+
+## 7. Error Handling
+
+| Scenario | Behaviour |
+|----------|-----------|
+| No recovery points | Orchestrator throws error (non-validation) |
+| Restore timeout | Error after 55m (FAILED not published) |
+| Validator throws | Orchestrator records FAILED with parse/message fallback |
+| Validator returns malformed JSON | Treated as FAILED with parse error message |
+
+## 8. Extensibility
+
+- Add multi-resource batch mode via Step Functions if needed.
+- Support additional resource types by adjusting Metadata mapping (e.g. RDS cluster restore specifics).
+- Emit custom metrics (future) for restore duration & validator latency.
+
+## 9. Example S3 Validator Patterns
+
+| Pattern | Description |
+|---------|-------------|
+| Key existence | Ensure enumerated critical objects are present (manifest-sourced) |
+| Non-empty bucket | Basic continuity signal after restore |
+| Minimum count | Validate approximate dataset size threshold |
+| Sample integrity | (Future) HEAD + ETag comparison against manifest |
+
+## 10. Terraform Surfaces
+
+Module `aws-backup-manual-validation` variables:
+
+- `backup_vault_name` (string, required)
+- `validation_lambda_arn` (string, required)
+- `resource_type` (string, e.g. S3)
+- `target_bucket_name` (string, S3 convenience)
+- `name_prefix` (string)
+
+Outputs:
+
+- `orchestrator_lambda_arn`
+
+## 11. Invocation Examples
+
+AWS CLI (invoke latest recovery point):
+
+```bash
+aws lambda invoke \
+  --function-name myproj-dev-manual-restore-orchestrator \
+  --payload '{}' out.json && cat out.json | jq
+```
+
+Explicit recovery point + expected keys:
+
+```bash
+aws lambda invoke \
+  --function-name myproj-dev-manual-restore-orchestrator \
+  --payload '{"recoveryPointArn":"arn:aws:backup:..","expectedKeys":["manifest.json","data/file1"]}' out.json
+```
+
+## 12. Limitations
+
+- Long-running restores may exceed Lambda timeout (convert to Step Functions for scale/time).
+- Only single resource restore per invocation.
+- No built-in notification channel (user can layer SNS or EventBridge rule on Lambda logs/exits).
+
+## 13. Future Enhancements
+
+- Step Functions wrapper for large parallel restores.
+- Parameter / Secrets retrieval for RDS validation credentials.
+- Config-driven validator selection registry.

examples/customer-s3-validator/index.ts

@@ -0,0 +1,58 @@
+import { S3Client, HeadObjectCommand, ListObjectsV2Command } from "@aws-sdk/client-s3";
+
+const s3 = new S3Client({});
+
+/* Example validator strategy:
+   1. If event.expectedKeys provided -> verify each exists.
+   2. Else if event.s3.bucket provided -> ensure bucket contains at least one object (or expectedMinObjects).
+   Return status + message summarising findings.
+*/
+
+interface EventShape {
+  restoreJobId: string;
+  recoveryPointArn: string;
+  resourceType: string;
+  createdResourceArn?: string;
+  targetBucket?: string;
+  s3?: { bucket?: string };
+  expectedKeys?: string[];
+  expectedMinObjects?: number;
+}
+
+export const handler = async (event: EventShape) => {
+  const bucket = event.targetBucket || event.s3?.bucket;
+  if (!bucket) {
+    return { status: "SKIPPED", message: "No bucket specified" };
+  }
+
+  if (event.expectedKeys && event.expectedKeys.length > 0) {
+    const missing: string[] = [];
+    for (const key of event.expectedKeys) {
+      try {
+        await s3.send(new HeadObjectCommand({ Bucket: bucket, Key: key }));
+      } catch (e) {
+        missing.push(key);
+      }
+    }
+    if (missing.length > 0) {
+      return { status: "FAILED", message: `Missing ${missing.length} objects`, missing };
+    }
+    return { status: "SUCCESSFUL", message: `All ${event.expectedKeys.length} expected objects present` };
+  }
+
+  // Fallback: simple non-empty check or min object threshold
+  const min = event.expectedMinObjects ?? 1;
+  let found = 0;
+  let ContinuationToken: string | undefined = undefined;
+  while (found < min) {
+    const resp = await s3.send(new ListObjectsV2Command({ Bucket: bucket, MaxKeys: 1000, ContinuationToken }));
+    const count = resp.Contents?.length || 0;
+    found += count;
+    if (!resp.IsTruncated) break;
+    ContinuationToken = resp.NextContinuationToken;
+  }
+  if (found < min) {
+    return { status: "FAILED", message: `Only ${found} objects found (< ${min})` };
+  }
+  return { status: "SUCCESSFUL", message: `Found ${found} objects (>= ${min})` };
+};

examples/customer-s3-validator/package.json

@@ -0,0 +1,16 @@
+{
+  "name": "customer-s3-validator-example",
+  "version": "0.1.0",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "build": "tsc -p tsconfig.json"
+  },
+  "dependencies": {
+    "@aws-sdk/client-s3": "^3.637.0"
+  },
+  "devDependencies": {
+    "typescript": "^5.4.0",
+    "@types/node": "^20.11.0"
+  }
+}

examples/customer-s3-validator/tsconfig.json

@@ -0,0 +1,14 @@
+{
+  "compilerOptions": {
+    "target": "ES2020",
+    "module": "ES2020",
+    "moduleResolution": "Node",
+    "outDir": "dist",
+    "rootDir": ".",
+    "esModuleInterop": true,
+    "strict": true,
+    "skipLibCheck": true
+  },
+  "include": ["index.ts"],
+  "exclude": ["node_modules"]
+}

examples/manual-validation/main.tf

@@ -0,0 +1,83 @@
+terraform {
+  required_version = ">= 1.5.0"
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = ">= 5.0"
+    }
+  }
+}
+
+provider "aws" {
+  region = var.region
+}
+
+variable "region" { type = string }
+variable "name_prefix" { type = string }
+variable "backup_vault_name" { type = string }
+variable "restore_bucket" { type = string }
+
+# Example customer validator lambda (upload dist bundle manually or integrate build pipeline).
+resource "aws_lambda_function" "customer_validator" {
+  function_name = "${var.name_prefix}-customer-s3-validator"
+  role          = aws_iam_role.customer_validator.arn
+  handler       = "index.handler"
+  runtime       = "nodejs20.x"
+  filename      = "./lambda_customer_validator.zip" # user supplied artifact
+  source_code_hash = filebase64sha256("./lambda_customer_validator.zip")
+  timeout       = 60
+  environment {
+    variables = {}
+  }
+}
+
+resource "aws_iam_role" "customer_validator" {
+  name               = "${var.name_prefix}-customer-s3-validator-role"
+  assume_role_policy = data.aws_iam_policy_document.lambda_assume.json
+}
+
+data "aws_iam_policy_document" "lambda_assume" {
+  statement {
+    actions = ["sts:AssumeRole"]
+    principals { type = "Service" identifiers = ["lambda.amazonaws.com"] }
+  }
+}
+
+resource "aws_iam_role_policy_attachment" "logs_attach_customer" {
+  role       = aws_iam_role.customer_validator.name
+  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
+}
+
+resource "aws_iam_policy" "customer_s3_policy" {
+  name   = "${var.name_prefix}-customer-s3-validator-policy"
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Effect   = "Allow"
+        Action   = ["s3:ListBucket", "s3:GetObject", "s3:HeadObject"]
+        Resource = [
+          "arn:aws:s3:::${var.restore_bucket}",
+          "arn:aws:s3:::${var.restore_bucket}/*"
+        ]
+      }
+    ]
+  })
+}
+
+resource "aws_iam_role_policy_attachment" "customer_validator_attach" {
+  role       = aws_iam_role.customer_validator.name
+  policy_arn = aws_iam_policy.customer_s3_policy.arn
+}
+
+module "manual_validation" {
+  source                = "../../modules/aws-backup-manual-validation"
+  enable                = true
+  name_prefix           = var.name_prefix
+  backup_vault_name     = var.backup_vault_name
+  resource_type         = "S3"
+  validation_lambda_arn = aws_lambda_function.customer_validator.arn
+  target_bucket_name    = var.restore_bucket
+}
+
+output "orchestrator_lambda" { value = module.manual_validation.orchestrator_lambda_arn }

modules/aws-backup-manual-validation/README.md

@@ -0,0 +1,84 @@
+# AWS Backup Manual Restore Validation Module
+
+Provides an on-demand Lambda **orchestrator** that:
+
+1. Selects a recovery point (latest by default) from a specified backup vault.
+2. Starts a restore job for the chosen recovery point (supports S3 in example).
+3. Waits for restore job completion (polling AWS Backup).
+4. Invokes a **customer-provided validation Lambda** (you own resource-specific logic).
+5. Publishes validation status back to AWS Backup using `PutRestoreValidationResult`.
+
+This pattern differs from automated restore testing plans: it is **manually triggered** (e.g. via `aws lambda invoke` or an API Gateway front-end) and delegates validation logic entirely to a customer-maintained Lambda.
+
+## Key Design Principles
+
+- **Separation of concerns**: Orchestrator handles restore lifecycle & result publishing; customer Lambda handles semantic integrity checks.
+- **Pluggable**: Any runtime or language for validator (only contract is JSON in/out).
+- **Minimal surface**: No Step Functions required for single-resource manual validation.
+
+## Orchestrator Environment Variables
+
+| Variable | Purpose |
+|----------|---------|
+| `BACKUP_VAULT_NAME` | Source vault to enumerate recovery points |
+| `RESOURCE_TYPE` | Backup resource type (e.g. `S3`) |
+| `VALIDATOR_LAMBDA` | ARN of customer validator Lambda |
+| `TARGET_BUCKET` | (S3 only) Destination bucket name to validate |
+| `RESTORE_ROLE_ARN` | (Optional) IAM role used for restore job |
+
+## Customer Validator Contract
+
+**Invocation Payload** (example):
+
+```json
+{
+  "restoreJobId": "1234abcd",
+  "recoveryPointArn": "arn:aws:backup:...:recovery-point:...",
+  "resourceType": "S3",
+  "createdResourceArn": "arn:aws:s3:::restored-bucket",
+  "targetBucket": "restored-bucket",
+  "s3": { "bucket": "restored-bucket" }
+}
+```
+
+**Return Object**:
+
+```json
+{ "status": "SUCCESSFUL|FAILED|SKIPPED", "message": "Human readable summary" }
+```
+Statuses are normalised by the orchestrator before calling `PutRestoreValidationResult`.
+
+## Terraform Inputs
+
+See `variables.tf` for full list. Essential:
+
+```hcl
+module "manual_validation" {
+  source               = "../modules/aws-backup-manual-validation"
+  enable               = true
+  name_prefix          = var.name_prefix
+  backup_vault_name    = var.backup_vault_name
+  resource_type        = "S3"
+  validation_lambda_arn = aws_lambda_function.customer_validator.arn
+  target_bucket_name   = var.target_restore_bucket
+}
+```
+
+## Example Validator (S3 Presence / Count)
+
+See `../../examples/customer-s3-validator` for a full TypeScript implementation scanning a set of expected keys or listing a prefix to ensure non-empty restore.
+
+## Operational Notes
+
+- Timeouts: Orchestrator Lambda default timeout is 15 minutes; long restores will exceed this—use small test datasets or adapt to Step Functions if needed.
+- Costs: Avoid listing millions of S3 keys in the validator; prefer sampling.
+- IAM Hardening: Current policy uses broad `backup:*` subset and `s3:Get*`; tighten to specific ARNs in production.
+
+## Future Enhancements
+
+- Option to specify explicit recovery point instead of auto-pick (supported already via event.recoveryPointArn field).
+- Emit custom CloudWatch metrics for validation duration & success rate.
+- Optional SNS notification on failure.
+
+---
+MIT style licensing per repository policy.