learn to run pg_dump & aws s3 cp on lambda

Summary

With this repo, I learned:

• layers on lambda, versioning them
• how to get custom runtime over lambda working
• blend all free stuffs on internet to play along
• storage classes on S3, pricing

Arch

• [ lambda ]
  • layer1: pg_dump runner (I named bash-runner, don’t know why)
  • layer2: awscli
  • script: ~main.sh~ with handler function calling other dependent functions
  • env: ~.env~ file housing the secrets, I didn’t know lambda also provides place to put env vars #reviselater
• pg_dump <– [pg@play-with-docker]:5432 via ngrok
  1. aws s3 cp dump.sql -> s3 bucket

Step for reproduction

you have been warned, please try this at home ;)

If you wanna do stupid thing like me here are you steps of reproduction:

1. Preparation:
   1. get free aws account by filling form at end
   2. get a pg running available public to world ( I am not doing lambda in vpc for now) some how, I will write about it if necessary. I managed to do it with Play with docker + ngrok fusion
   3. configure things needed for lambda {iam role}, & s3 bucket(create new if necessary, and allow lambda role to write to it). For sake of simplicity, I gave full control for time being. (lambda full, and s3fullaccess)
2. Execution:
   1. create a lambda layers
      1. layer for pg_dump
      2. layer for awscli

   lol, I revised this,we have a Makefile now :) make build_layers

3. Deployment
   • create lambda layers

     

   • deploy lambda function(main.sh)

     

   I have a recorded my screen for demo, I will share later.

Useful Resources:

• How to use AWS CLI within a Lambda function (aws s3 sync from Lambda)
• Getting Started with AWS Lambda and Serverless Computing - AWS Online Tech Talks
• Genese Cloud Academy free $75 credit, fill the form at end if you want to play with aws for “free”
• Play with docker + ngrok fusion; use Linux (32-bit) on alpine
• Postgresql
  • sample db: https://github.com/h8/employees-database

Meta

I need a pg quick

quick_pg(){
    cat > /tmp/secrets <<EOF
POSTGRES_DB=mydb
POSTGRES_USER=my_user
POSTGRES_PASSWORD=uGbwtA76uXD2
EOF
    docker run --rm -it --env-file /tmp/secrets -p5432:5432 postgres:11-alpine
}