From 6d24247ce01c940e3a0c69eb4346896f16aff23b Mon Sep 17 00:00:00 2001 From: Mike Lonergan Date: Sun, 5 May 2019 15:40:30 -0700 Subject: [PATCH 1/6] Re-organizing the 2018-era database infrastructure instructions Reduce the odds that folks think this is our 2019-era database implementation --- ...create-backup-for-new-database-creation.md | 52 ++++++++++++++ ...ebuild-the-centralized-database-service.md | 68 +++++++++++++++++++ 2 files changed, 120 insertions(+) create mode 100644 docs/database/2018/HOWTO-create-backup-for-new-database-creation.md create mode 100644 docs/database/2018/HOWTO-rebuild-the-centralized-database-service.md diff --git a/docs/database/2018/HOWTO-create-backup-for-new-database-creation.md b/docs/database/2018/HOWTO-create-backup-for-new-database-creation.md new file mode 100644 index 0000000..fe8df5a --- /dev/null +++ b/docs/database/2018/HOWTO-create-backup-for-new-database-creation.md @@ -0,0 +1,52 @@ +# HOWTO create a backup for new database creation + +When standing up a new database instance, the project team must submit that data in a specified format so that the database automation we support can ingest the initial data load. The currently-supported procedures for initial data load are [here](https://github.com/hackoregon/civic-devops/blob/master/docs/HOWTO-rebuild-the-centralized-database-service.md#restore-databases-from-backup) + +## Requirements + +These are the current requirements for submitting a database backup to the Hack Oregon DevOps squad that can be successfully used to perform the initial load of a new database instance. + +1. Coordinate with the Hack Oregon DevOps squad to finalize the name of the database instance and acquire the username that will own that database instance (e.g. a project named `mars-space-flight` might have two databases named `mars-space-flight-surface-lander` and `mars-space-flight-liftoff`). + +2. The current required format for database backups is compressed plain-text, and the current supported version of PostgreSQL at this time (March 2018) is 9.6.6 (latest supported on Amazon Linux 2). Other formats are sometime incompatible with PostgreSQL servers at the same release or newer. Note that database backups are not designed to be restored to older versions of PostgreSQL - e.g. don't take a backup from 10.3 and expect it to restore to 9.6. + + A plain-text backup is SQL code with some `psql` commands mixed in. You can read the backup's contents with a text editor - a compressed plain text backup can be read if you simply uncompress it first. + +3. A database on the source server should have the same owner name as it will have on the destination server. All the objects in the database should have that owner name too. This means the user / role must exist on the source when the backup is created and on the destination before it is restored. + + You can change owners of databases and the objects they contain with SQL script like the following - e.g. source database is `odot_crash_data`, original owner is `znmeb`, new owner is `transportation-systems`: + + ``` + ALTER DATABASE odot_crash_data OWNER TO "transportation-systems"; + REASSIGN OWNED BY znmeb TO "transportation-systems"; + ``` + + Note: the double quotes are required because of the hyphen in the new owner's name. + +4. This command will create a compressed plain-text backup: + + ``` + pg_dump -Fp -v -C -c --if-exists -d \ + | gzip -c > .sql.gz + ``` + + `` is the database. Run this as the database superuser `postgres` on Linux. The parameters: + * `-Fp`: plain text format + * `-v`: verbose + * `-C -c`: create a clean new database. This is done by DROPping the database objects. If they doesn't exist, the DROP will error, so ... + * `--if-exists`: don't DROP if it doesn't exist. You'll get a `NOTICE` instead of an `ERROR` and the restore will continue! + +5. Command to restore the compressed backup: + + ``` + gzip -dc .sql.gz | psql -v ON_ERROR_STOP=1 + ``` + + Run this as the database superuser `postgres`. As noted above, the owners of all the objects on the backup file must exist in the destination server or the restore will fail. + +6. Did it work? Usually you'll see `Restore completed` at the end of a successful restore. The `ON_ERROR_STOP=1` option will force `psql` to stop on its first error. + +7. The latest release of `data-science-pet-containers` has an option to do restore testing on an Amazon Linux 2 container running PostgreSQL from Amazon Linux Extras. See + +## Need Help? +If you need help generating compatible backups from your development data, please post your questions to the #chat-databases channel on the [Hack Oregon 2018 Slack group](https://hackoregon2018.slack.com). diff --git a/docs/database/2018/HOWTO-rebuild-the-centralized-database-service.md b/docs/database/2018/HOWTO-rebuild-the-centralized-database-service.md new file mode 100644 index 0000000..3349011 --- /dev/null +++ b/docs/database/2018/HOWTO-rebuild-the-centralized-database-service.md @@ -0,0 +1,68 @@ +# HOWTO: Rebuild the Centralized Database service + +This set of scripts and procedures is how we built the PostgreSQL database service for the 2018 Hack Oregon projects. + +1. Build an EC2 machine +2. Install and configure PostgreSQL +3. Create users and databases +4. Restore databases from backup +5. Extend the EBS data volume for new databases + +## Build an EC2 machine + +This procedure relies on the `create-ec2-machine-database.sh` script: + +* Prereq: run the script from an environment where AWS keys are available and the keys grant EC2 machine creation privileges +* Prereq: the [awscli](https://docs.aws.amazon.com/cli/latest/userguide/installing.html) is installed +* Step 1: edit any of the VARIABLES to match your intended EC2 virtual machine environment, especially the KEYNAME +* Step 2: run `./create-ec2-machine-database.sh NAME_OF_RESULTING_MACHINE`, where NAME_OF_RESULTING_MACHINE will be used to populate the AWS "Name" tag of the machine + +## Install and configure PostgreSQL + +NOTE: "development" is meant to signify this was the build for a development, not a production, instance of the database. +This procedure relies on the `create-database-development.sh` script that runs on the server: + +* Prereq: an EC2 machine has already been built +* Assumption: no database service has been installed or configured on the machine +* Prereq: the AWS "amazon-linux-extras" repo is available, and the version of PostgreSQL installed is the Amazon Linux package +* Step 1: edit any of the VARIABLES to match your intended PostgreSQL service and environment +* Step 2: run `./create-database-development.sh` with no parameters +* Step 3: fill in the `postgres` database user password when prompted + +## Create users and databases + +This procedure relies on the `create-db.sh` script that runs on the PostgreSQL server + +* Prereq: PostgreSQL has been configured successfully and is running +* Prereq: run the script from within the machine that hosts the targeted PostgreSQL service +* Prereq: run the script from a shell where `sudo -u postgres` will silently succeed +* Step 1: decide upon the name of the database instance, name of database user who will own that instance, and password for that user +* Step 2: `scp` the script to the server and `ssh` in as `ec2-user` (or equivalent) +* Step 3: run `./create-db.sh` with no parameters (or alternatively, feed the values from Step 1 in as the correct parameters) +* Step 4: fill in the database user password when prompted (if running interactively) +* Result: database user and associated database are created + +## Restore databases from backup + +This procedure relies on stepwise commands run on the PostgreSQL server to restore database data to an empty database instance. + +There are two scenarios: `.backup` file that was generated from `pgdump` native format, or `.sql.gz` file that was generated from `pgdump` in "compressed SQL" format + +### Restore from .sql.gz + +* Prereq: database instance has been created +* Prereq: named database user is owner of the target database instance +* Prereq: backup has been created from instance with the same database user name and database instance name +* Step 1: `ssh` into the machine hosting the PostgreSQL service as `ec2-user` (or equivalent) +* Step 2: run wget to download the backup file e.g. `wget https://s3-us-west-2.amazonaws.com/hacko-data-archive/2018-transportation-systems/data/interim/passenger_census.backup` +* Step 3: run `gzip -dc BACKUP_FILE.sql.gz | sudo -u postgres psql` +* Step 4: test that the restore succeeded (i.e. the database is not still empty) by running `sudo -u postgres psql -d [DATABASE_INSTANCE] -c 'SELECT COUNT(*) FROM [DATABASE_INSTANCE/TABLE_NAME];'` where [DATABASE_INSTANCE] is the database instance name e.g. `sudo -u postgres psql -d passenger_census -c 'SELECT COUNT(*) FROM passenger_census;'` +* Note: the test command should result in a non-zero count + +# Extend the EBS data volume for new databases + +EBS volumes are allocated statically for EC2 machines - that is, if you need to allow your data files to grow to 500GB, then you need to explicitly allocate 500GB to the EBS volume housing those data files. + +Since EBS space isn't free, Hack Oregon is working to allocate only as much space as is needed for the existing databases. When a new database instance is added, chances are the EBS volume needs to be "grown" to accommodate the new space requirements. + +Detailed instructions on how to [increase the space on the data volume can be found here](https://github.com/hackoregon/civic-devops/blob/master/docs/HOWTO-extend-EBS-Volume-size.md). From 6761156025cfd0894101e770918c4a72042bb9e4 Mon Sep 17 00:00:00 2001 From: Mike Lonergan Date: Sun, 5 May 2019 15:43:40 -0700 Subject: [PATCH 2/6] Remove the 2018 implementation docs from their old location --- ...create-backup-for-new-database-creation.md | 52 -------------- ...ebuild-the-centralized-database-service.md | 68 ------------------- 2 files changed, 120 deletions(-) delete mode 100644 docs/HOWTO-create-backup-for-new-database-creation.md delete mode 100644 docs/HOWTO-rebuild-the-centralized-database-service.md diff --git a/docs/HOWTO-create-backup-for-new-database-creation.md b/docs/HOWTO-create-backup-for-new-database-creation.md deleted file mode 100644 index fe8df5a..0000000 --- a/docs/HOWTO-create-backup-for-new-database-creation.md +++ /dev/null @@ -1,52 +0,0 @@ -# HOWTO create a backup for new database creation - -When standing up a new database instance, the project team must submit that data in a specified format so that the database automation we support can ingest the initial data load. The currently-supported procedures for initial data load are [here](https://github.com/hackoregon/civic-devops/blob/master/docs/HOWTO-rebuild-the-centralized-database-service.md#restore-databases-from-backup) - -## Requirements - -These are the current requirements for submitting a database backup to the Hack Oregon DevOps squad that can be successfully used to perform the initial load of a new database instance. - -1. Coordinate with the Hack Oregon DevOps squad to finalize the name of the database instance and acquire the username that will own that database instance (e.g. a project named `mars-space-flight` might have two databases named `mars-space-flight-surface-lander` and `mars-space-flight-liftoff`). - -2. The current required format for database backups is compressed plain-text, and the current supported version of PostgreSQL at this time (March 2018) is 9.6.6 (latest supported on Amazon Linux 2). Other formats are sometime incompatible with PostgreSQL servers at the same release or newer. Note that database backups are not designed to be restored to older versions of PostgreSQL - e.g. don't take a backup from 10.3 and expect it to restore to 9.6. - - A plain-text backup is SQL code with some `psql` commands mixed in. You can read the backup's contents with a text editor - a compressed plain text backup can be read if you simply uncompress it first. - -3. A database on the source server should have the same owner name as it will have on the destination server. All the objects in the database should have that owner name too. This means the user / role must exist on the source when the backup is created and on the destination before it is restored. - - You can change owners of databases and the objects they contain with SQL script like the following - e.g. source database is `odot_crash_data`, original owner is `znmeb`, new owner is `transportation-systems`: - - ``` - ALTER DATABASE odot_crash_data OWNER TO "transportation-systems"; - REASSIGN OWNED BY znmeb TO "transportation-systems"; - ``` - - Note: the double quotes are required because of the hyphen in the new owner's name. - -4. This command will create a compressed plain-text backup: - - ``` - pg_dump -Fp -v -C -c --if-exists -d \ - | gzip -c > .sql.gz - ``` - - `` is the database. Run this as the database superuser `postgres` on Linux. The parameters: - * `-Fp`: plain text format - * `-v`: verbose - * `-C -c`: create a clean new database. This is done by DROPping the database objects. If they doesn't exist, the DROP will error, so ... - * `--if-exists`: don't DROP if it doesn't exist. You'll get a `NOTICE` instead of an `ERROR` and the restore will continue! - -5. Command to restore the compressed backup: - - ``` - gzip -dc .sql.gz | psql -v ON_ERROR_STOP=1 - ``` - - Run this as the database superuser `postgres`. As noted above, the owners of all the objects on the backup file must exist in the destination server or the restore will fail. - -6. Did it work? Usually you'll see `Restore completed` at the end of a successful restore. The `ON_ERROR_STOP=1` option will force `psql` to stop on its first error. - -7. The latest release of `data-science-pet-containers` has an option to do restore testing on an Amazon Linux 2 container running PostgreSQL from Amazon Linux Extras. See - -## Need Help? -If you need help generating compatible backups from your development data, please post your questions to the #chat-databases channel on the [Hack Oregon 2018 Slack group](https://hackoregon2018.slack.com). diff --git a/docs/HOWTO-rebuild-the-centralized-database-service.md b/docs/HOWTO-rebuild-the-centralized-database-service.md deleted file mode 100644 index 3349011..0000000 --- a/docs/HOWTO-rebuild-the-centralized-database-service.md +++ /dev/null @@ -1,68 +0,0 @@ -# HOWTO: Rebuild the Centralized Database service - -This set of scripts and procedures is how we built the PostgreSQL database service for the 2018 Hack Oregon projects. - -1. Build an EC2 machine -2. Install and configure PostgreSQL -3. Create users and databases -4. Restore databases from backup -5. Extend the EBS data volume for new databases - -## Build an EC2 machine - -This procedure relies on the `create-ec2-machine-database.sh` script: - -* Prereq: run the script from an environment where AWS keys are available and the keys grant EC2 machine creation privileges -* Prereq: the [awscli](https://docs.aws.amazon.com/cli/latest/userguide/installing.html) is installed -* Step 1: edit any of the VARIABLES to match your intended EC2 virtual machine environment, especially the KEYNAME -* Step 2: run `./create-ec2-machine-database.sh NAME_OF_RESULTING_MACHINE`, where NAME_OF_RESULTING_MACHINE will be used to populate the AWS "Name" tag of the machine - -## Install and configure PostgreSQL - -NOTE: "development" is meant to signify this was the build for a development, not a production, instance of the database. -This procedure relies on the `create-database-development.sh` script that runs on the server: - -* Prereq: an EC2 machine has already been built -* Assumption: no database service has been installed or configured on the machine -* Prereq: the AWS "amazon-linux-extras" repo is available, and the version of PostgreSQL installed is the Amazon Linux package -* Step 1: edit any of the VARIABLES to match your intended PostgreSQL service and environment -* Step 2: run `./create-database-development.sh` with no parameters -* Step 3: fill in the `postgres` database user password when prompted - -## Create users and databases - -This procedure relies on the `create-db.sh` script that runs on the PostgreSQL server - -* Prereq: PostgreSQL has been configured successfully and is running -* Prereq: run the script from within the machine that hosts the targeted PostgreSQL service -* Prereq: run the script from a shell where `sudo -u postgres` will silently succeed -* Step 1: decide upon the name of the database instance, name of database user who will own that instance, and password for that user -* Step 2: `scp` the script to the server and `ssh` in as `ec2-user` (or equivalent) -* Step 3: run `./create-db.sh` with no parameters (or alternatively, feed the values from Step 1 in as the correct parameters) -* Step 4: fill in the database user password when prompted (if running interactively) -* Result: database user and associated database are created - -## Restore databases from backup - -This procedure relies on stepwise commands run on the PostgreSQL server to restore database data to an empty database instance. - -There are two scenarios: `.backup` file that was generated from `pgdump` native format, or `.sql.gz` file that was generated from `pgdump` in "compressed SQL" format - -### Restore from .sql.gz - -* Prereq: database instance has been created -* Prereq: named database user is owner of the target database instance -* Prereq: backup has been created from instance with the same database user name and database instance name -* Step 1: `ssh` into the machine hosting the PostgreSQL service as `ec2-user` (or equivalent) -* Step 2: run wget to download the backup file e.g. `wget https://s3-us-west-2.amazonaws.com/hacko-data-archive/2018-transportation-systems/data/interim/passenger_census.backup` -* Step 3: run `gzip -dc BACKUP_FILE.sql.gz | sudo -u postgres psql` -* Step 4: test that the restore succeeded (i.e. the database is not still empty) by running `sudo -u postgres psql -d [DATABASE_INSTANCE] -c 'SELECT COUNT(*) FROM [DATABASE_INSTANCE/TABLE_NAME];'` where [DATABASE_INSTANCE] is the database instance name e.g. `sudo -u postgres psql -d passenger_census -c 'SELECT COUNT(*) FROM passenger_census;'` -* Note: the test command should result in a non-zero count - -# Extend the EBS data volume for new databases - -EBS volumes are allocated statically for EC2 machines - that is, if you need to allow your data files to grow to 500GB, then you need to explicitly allocate 500GB to the EBS volume housing those data files. - -Since EBS space isn't free, Hack Oregon is working to allocate only as much space as is needed for the existing databases. When a new database instance is added, chances are the EBS volume needs to be "grown" to accommodate the new space requirements. - -Detailed instructions on how to [increase the space on the data volume can be found here](https://github.com/hackoregon/civic-devops/blob/master/docs/HOWTO-extend-EBS-Volume-size.md). From 21492409524ed67860962f25da99b713407edf2c Mon Sep 17 00:00:00 2001 From: Mike Lonergan Date: Sun, 5 May 2019 16:59:25 -0700 Subject: [PATCH 3/6] Moved the instrux for DB readonly user These instructions should be good for both 2018 and 2019 so I'll put them here --- docs/{ => database}/HOWTO-enable-readonly-database-user.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/{ => database}/HOWTO-enable-readonly-database-user.md (100%) diff --git a/docs/HOWTO-enable-readonly-database-user.md b/docs/database/HOWTO-enable-readonly-database-user.md similarity index 100% rename from docs/HOWTO-enable-readonly-database-user.md rename to docs/database/HOWTO-enable-readonly-database-user.md From 3fad77535010f944ffa59d9abf9f12e9fbae2401 Mon Sep 17 00:00:00 2001 From: Mike Lonergan Date: Sun, 5 May 2019 17:00:33 -0700 Subject: [PATCH 4/6] Initial instructions documenting the RDS instance creation process This models the manual process for creating individual RDS instances. Automation of these instructions in CloudFormation will be next up for someone. --- docs/database/2019/HOWTO: Build RDS instances | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 docs/database/2019/HOWTO: Build RDS instances diff --git a/docs/database/2019/HOWTO: Build RDS instances b/docs/database/2019/HOWTO: Build RDS instances new file mode 100644 index 0000000..fa30712 --- /dev/null +++ b/docs/database/2019/HOWTO: Build RDS instances @@ -0,0 +1,43 @@ +# HOWTO Build RDS instances (for 2019 project season) + +Summary: these are the manual RDS-instantiation instrux that will form the basis for CloudFormation automation of each 2019 project's databases. + +## Background +In the 2019 Hack Oregon project season, we will be using per-project RDS instances for hosting both the staging and production databases. + +Per-project - rather than a centralized database instance shared by all databases - because at the outset of the 2019 project season, there was more uncertainty than usual about the size or compute-resource requirements for each 2019 project. + +Staging vs. Production - rather than a single layer for full-season usage - because we assume that we'll get around to the laborious and expensive task of standing up a production-ready, well-protected API-to-DB layer - a less-free-for-all infrastructure for Django-to-DB compute than we've had in the past (despite all good intentions). + +## Procedure + +1. Create an RDS instance with the following shared characteristics (suitable for staging aka "development" work): +- - engine = PostgreSQL +- - Use Case = dev/test +- - DB engine version = PostgreSQL 11.2-R1 +- - DB instance class = db.t2.small +- - Multi-AZ deployment = No +- - Storage type = General purpose (SSD) +- - Allocated storage = 100 GB +- - Virtual Private Cloud = Default VPC +- - Public accessibility = yes (note: will be "no" for production DBs, that can only be accessed via Django container in ECS) +- - Availability zone = no preference +- - VPC security groups = hacko-public-database (note: will be more restrictive for production DBs) +- - IAM DB authentication = Disable +- - Encryption = Enable +- - Master key = (default) aws/rds +- - Backup retention period = 7 days +- - Backup window = no preference +- - Enhanced monitoring = Enable enhanced monitoring +- - Performance insights = Enable performance insights, retention period = 7 days, Master key = (default) aws/rds +- - Log exports = Postgresql log +- - Auto minor version upgrade = Enable auto minor version upgrade +- - Maintenance window = No preference +- - Deletion protection = Enable delete protection +1. Create a login role according to the naming conventions e.g. `transportation2019` - which has following privileges: Can login, Inherit rights from the parent roles +1. Create the database according to the https://docs.google.com/spreadsheets/d/147thL899Bf8IL3ma1S9XBrNXL2xYsIRM5mE-3fceIcQ/ naming scheme +1. Assign the created login role as the Owner of the DB +- - note: in AWS RDS, the role creating the database must have the role which will be owner of the database being created: https://stackoverflow.com/a/34898033 +1. Execute the following command in the new DB using the Query Tool or other SQL automation: +`CREATE EXTENSION postgis;` +1. If you need add another role with read-only privileges to the DB, use the script in *HOWTO-enable-readonly-database-user* or similar. From be6ebf564466fc31365f628f4b72250bd4cfc48d Mon Sep 17 00:00:00 2001 From: Mike Lonergan Date: Sun, 19 May 2019 11:50:09 -0700 Subject: [PATCH 5/6] Procedure for using jumpbox for 2019 DB restore --- ... Use jumpbox to restore PostgreSQL from S3.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 docs/database/2019/HOWTO: Use jumpbox to restore PostgreSQL from S3.md diff --git a/docs/database/2019/HOWTO: Use jumpbox to restore PostgreSQL from S3.md b/docs/database/2019/HOWTO: Use jumpbox to restore PostgreSQL from S3.md new file mode 100644 index 0000000..852be41 --- /dev/null +++ b/docs/database/2019/HOWTO: Use jumpbox to restore PostgreSQL from S3.md @@ -0,0 +1,16 @@ +# HOWTO: use the "DB jumpbox" to restore PostgreSQL database from S3 + +Summary: SSH into the jumpbox, grab your backup file, restore your backup. + +## Steps + +1. *Get the keys*: Talk to #team-infra on our Slack group to get a copy of the SSH keys (in `.pem` format) +2. *Place the keys*: Download the .pem file then run `mv hackoregon-2019-db-restore-jumpbox.pem ~/.ssh` +3. *Secure the keys*: Run `chmod 400 ~/.ssh/hackoregon-2019-db-restore-jumpbox.pem` +4. *Use the keys to get into the box*: Run `ssh -i ~/.ssh/hackoregon-2019-db-restore-jumpbox.pem ec2-user@ec2-34-220-186-62.us-west-2.compute.amazonaws.com` to get into the jumpbox +5. *Get the backup file*: Copy your backup from the S3 location with a command of the form aws s3 cp s3://hacko-data-archive/(team folder)/(backups folder)/(backup file name) /backups +e.g. s3://hacko-data-archive/2017-team-budget/database-backup/budget.sql.gz /backups +6. *Restore your backup*: Run whatever command(s) you need to restore your database/data to your team's RDS instance for your team +7. *Cleanup your backup*: rm -rf /backups/*.* + +Your restore procedure will vary depending on the backup tools you're using, the kind of backup (e.g. data, schema or to-be-converted CSV) and where it's headed. From 2f23fcc49deabbd772271b382a46928dbab8d651 Mon Sep 17 00:00:00 2001 From: Mike Lonergan Date: Sun, 19 May 2019 12:07:12 -0700 Subject: [PATCH 6/6] misc uninteresting changes Hah, reverse-psyche'd you, these really are uninteresting --- bin/create-db.sh | 3 +- bin/upload-and-ssh-into-ec2.sh | 3 +- docs/database/2019/HOWTO: Build RDS instances | 43 ------------------- 3 files changed, 4 insertions(+), 45 deletions(-) delete mode 100644 docs/database/2019/HOWTO: Build RDS instances diff --git a/bin/create-db.sh b/bin/create-db.sh index 98eff9a..5f9c97d 100755 --- a/bin/create-db.sh +++ b/bin/create-db.sh @@ -32,7 +32,8 @@ if [ "$#" -eq 3 ]; then sudo -u postgres createuser --encrypted --no-createdb --no-createrole --no-superuser --no-replication $USERNAME sudo -u postgres psql -c ${SET_PWD_COMMAND} elif [ "$#" -eq 0 ]; then - # Running interatively (no arguments) + # Running interactively (no arguments) + echo "Enter the name of the database and username of that database's owner" echo 'NOTE: the database naming convention is (without bracket): -' read -p "Database name: " DB_NAME read -p "Username: " USERNAME diff --git a/bin/upload-and-ssh-into-ec2.sh b/bin/upload-and-ssh-into-ec2.sh index ea570ac..5b0a74e 100755 --- a/bin/upload-and-ssh-into-ec2.sh +++ b/bin/upload-and-ssh-into-ec2.sh @@ -5,7 +5,8 @@ EC2MACHINE=$1 # Public DNS of the EC2 VM e.g. ec2-54-200-42-122.us-west-2.compute.amazonaws.com FILETOUPLOAD="*" # PEMFILE=$2 -PEMFILE="~/.ssh/hackoregon-2018-database-dev-env.pem" +# PEMFILE="~/.ssh/hackoregon-2018-database-dev-env.pem" +PEMFILE="~/.ssh/hackoregon-integration-ecs.pem" scp -i $PEMFILE $FILETOUPLOAD ec2-user@${EC2MACHINE}:~ ssh -i $PEMFILE ec2-user@${EC2MACHINE} diff --git a/docs/database/2019/HOWTO: Build RDS instances b/docs/database/2019/HOWTO: Build RDS instances deleted file mode 100644 index fa30712..0000000 --- a/docs/database/2019/HOWTO: Build RDS instances +++ /dev/null @@ -1,43 +0,0 @@ -# HOWTO Build RDS instances (for 2019 project season) - -Summary: these are the manual RDS-instantiation instrux that will form the basis for CloudFormation automation of each 2019 project's databases. - -## Background -In the 2019 Hack Oregon project season, we will be using per-project RDS instances for hosting both the staging and production databases. - -Per-project - rather than a centralized database instance shared by all databases - because at the outset of the 2019 project season, there was more uncertainty than usual about the size or compute-resource requirements for each 2019 project. - -Staging vs. Production - rather than a single layer for full-season usage - because we assume that we'll get around to the laborious and expensive task of standing up a production-ready, well-protected API-to-DB layer - a less-free-for-all infrastructure for Django-to-DB compute than we've had in the past (despite all good intentions). - -## Procedure - -1. Create an RDS instance with the following shared characteristics (suitable for staging aka "development" work): -- - engine = PostgreSQL -- - Use Case = dev/test -- - DB engine version = PostgreSQL 11.2-R1 -- - DB instance class = db.t2.small -- - Multi-AZ deployment = No -- - Storage type = General purpose (SSD) -- - Allocated storage = 100 GB -- - Virtual Private Cloud = Default VPC -- - Public accessibility = yes (note: will be "no" for production DBs, that can only be accessed via Django container in ECS) -- - Availability zone = no preference -- - VPC security groups = hacko-public-database (note: will be more restrictive for production DBs) -- - IAM DB authentication = Disable -- - Encryption = Enable -- - Master key = (default) aws/rds -- - Backup retention period = 7 days -- - Backup window = no preference -- - Enhanced monitoring = Enable enhanced monitoring -- - Performance insights = Enable performance insights, retention period = 7 days, Master key = (default) aws/rds -- - Log exports = Postgresql log -- - Auto minor version upgrade = Enable auto minor version upgrade -- - Maintenance window = No preference -- - Deletion protection = Enable delete protection -1. Create a login role according to the naming conventions e.g. `transportation2019` - which has following privileges: Can login, Inherit rights from the parent roles -1. Create the database according to the https://docs.google.com/spreadsheets/d/147thL899Bf8IL3ma1S9XBrNXL2xYsIRM5mE-3fceIcQ/ naming scheme -1. Assign the created login role as the Owner of the DB -- - note: in AWS RDS, the role creating the database must have the role which will be owner of the database being created: https://stackoverflow.com/a/34898033 -1. Execute the following command in the new DB using the Query Tool or other SQL automation: -`CREATE EXTENSION postgis;` -1. If you need add another role with read-only privileges to the DB, use the script in *HOWTO-enable-readonly-database-user* or similar.