Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy CloudFront distributions in front of CopWatch #307

Merged
merged 57 commits into from
Jan 14, 2025
Merged
Changes from 1 commit
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
ee2c50d
deploy staging CDN
copelco Oct 21, 2024
9736978
Update documentation, remove django-cache-machine and references, com…
ronardcaktus Oct 23, 2024
3565f94
Return django-cache-machine and add comment
ronardcaktus Oct 23, 2024
8ff78a7
Remove mime-support
ronardcaktus Oct 24, 2024
a6fe60d
remove django-cache-machine from project(including old migration)
ronardcaktus Oct 24, 2024
00dbc99
upgrade dev requirements
copelco Oct 24, 2024
e2dc8f8
Remove cache_page decorator
ronardcaktus Oct 24, 2024
a9ee660
Merge branch 'CU-86897a79g-cache-misses' of github.com:caktus/Traffic…
copelco Oct 27, 2024
a5de3c0
rework cache primer to use celery tasks
copelco Oct 27, 2024
32cf120
upgrade dev/test Python reqs to latest versions
copelco Oct 27, 2024
5bb91fe
test get_group_urls
copelco Oct 27, 2024
ae8077b
add cf distro id to staging; support basicauth
copelco Oct 28, 2024
27da1d9
Allow key-less access to create CloudFront invalidations
copelco Oct 28, 2024
800d196
run 2 worker pods (one on each node)
copelco Oct 28, 2024
aee1624
Log request and response headers
copelco Oct 28, 2024
5bb0507
log all of request.META
copelco Oct 28, 2024
6efcbb7
print as json for now
copelco Oct 28, 2024
6e267c3
add option to limit primer by agency
copelco Oct 29, 2024
6af04be
allow ALLOWED_HOSTS to pull from dev env
copelco Oct 29, 2024
83f7c39
Refactor prime_cache.py to wait for CloudFront invalidation completion
copelco Oct 29, 2024
481c08e
Raise alert if request exceeds CF response timeout
copelco Oct 29, 2024
69f337e
auto retry prime task
copelco Oct 29, 2024
509f22d
set Accept header in session
copelco Oct 29, 2024
1d4aae4
add initial cache primer tests
copelco Oct 29, 2024
f7ad9be
add debugging notebook
copelco Oct 29, 2024
d36a942
document how to deploy the cf distro
copelco Oct 29, 2024
cab6699
re-enable cache primer
copelco Oct 29, 2024
5d8b9d6
fix circular import
copelco Oct 29, 2024
c626c7e
add additional debugging info for cache priming
copelco Oct 29, 2024
1995e2d
time group too
copelco Oct 29, 2024
fddaa74
Cache URLs with ? too
copelco Oct 30, 2024
dd804b3
prime additional endpoints
copelco Oct 30, 2024
81f0366
add indexes
copelco Oct 30, 2024
2e65ce6
pre-commit fix
copelco Oct 30, 2024
f85ca8c
adjust headers to try and match browser
copelco Oct 30, 2024
8d0e606
add mime-support package to try and fix svg content types
copelco Nov 1, 2024
61885cb
default cutoff = 0
copelco Nov 1, 2024
b00b44f
pre-commit fix
copelco Nov 1, 2024
7dd247a
upgrade pre-commit
copelco Nov 1, 2024
59aae0d
switch to httpx for http2 support
copelco Nov 1, 2024
7da6977
fix accept header
copelco Nov 2, 2024
c99b17b
add httpx timeout
copelco Nov 2, 2024
480e028
log response in separate middleware
copelco Nov 2, 2024
04cda6d
add Accept Language to primer; also prime with gzip header
copelco Nov 4, 2024
8f73d2e
refresh ContrabandSummary too
copelco Nov 4, 2024
0070905
support multiple cf stacks
copelco Nov 4, 2024
bec7e64
Resize DB instance to db.t4g.xlarge
ronardcaktus Nov 6, 2024
68478f9
Merge branch 'CU-86897a79g-cache-misses' of github.com:caktus/Traffic…
ronardcaktus Nov 6, 2024
7fd5b58
Merge branch 'CU-86897a79g-cache-misses' of github.com:caktus/Traffic…
copelco Nov 6, 2024
84b1012
Resize DB instance to db.t4g.xlarge
ronardcaktus Nov 6, 2024
9842ba7
link to caktus.aws-web-stacks ansible-upgrade branch
copelco Nov 6, 2024
5b78cc0
Merge branch 'CU-86897a79g-cache-misses' of github.com:caktus/Traffic…
copelco Nov 6, 2024
decfe90
add updated template with new cache policy
copelco Nov 21, 2024
96b58f7
upgrade caktus.aws-web-stacks to v0.3.0
copelco Nov 21, 2024
c2b0b39
remove logging middleware
copelco Jan 14, 2025
059ba88
update postgres client in tests
copelco Jan 14, 2025
969fc09
don't update to latest npm
copelco Jan 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
deploy staging CDN
Co-authored-by: Ronard <ronardcaktus@users.noreply.github.com>
copelco and ronardcaktus committed Oct 21, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit ee2c50d1b90b9e93b478118bb9232cd842b7554b
21 changes: 21 additions & 0 deletions deploy/deploy-cf-stack.yml
Original file line number Diff line number Diff line change
@@ -15,3 +15,24 @@
unexpectedly), execute the changeset. When complete, hit RETURN to continue or Control-C,
then A to abort.
when: cloudformation_stack_result is changed

- name: kubernetes cluster management
hosts: cluster
gather_facts: false
tags: cdn
vars:
ansible_connection: local
ansible_python_interpreter: "{{ ansible_playbook_python }}"
vars_files:
- group_vars/cdn.yml
roles:
- role: caktus.aws-web-stacks
tasks:
- name: Prompt if CloudFormation stack changed
pause:
prompt: >
A changeset was created for the CloudFormation stack. You MUST login to the AWS console,
inspect the changeset, and if everything looks good (i.e., it's not DELETING anything
unexpectedly), execute the changeset. When complete, hit RETURN to continue or Control-C,
then A to abort.
when: cloudformation_stack_result is changed
66 changes: 33 additions & 33 deletions deploy/group_vars/all.yml
Original file line number Diff line number Diff line change
@@ -25,39 +25,39 @@ admin_database_password: !vault |

stack_name: "{{ app_name }}-stack"

cloudformation_stack:
region: "{{ aws_region }}"
stack_name: "{{ stack_name }}"
template_bucket: "aws-web-stacks-{{ app_name }}"
# Webstacks required variable
template_local_path: '{{ playbook_dir + "/stack/eks-no-nat.yml" }}'
create_changeset: true
termination_protection: true

template_parameters:
PrimaryAZ: "{{ aws_region }}a"
SecondaryAZ: "{{ aws_region }}b"
DesiredScale: 2
MaxScale: 4
UseAES256Encryption: "true"
CustomerManagedCmkArn: ""
ContainerInstanceType: t3a.large
ContainerVolumeSize: 40
DatabaseAllocatedStorage: 100
DatabaseClass: db.t3.large
DatabaseEngineVersion: "12"
DatabaseParameterGroupFamily: postgres12
DatabaseMultiAZ: "false"
DatabaseUser: "{{ app_name }}_admin"
DatabasePassword: "{{ admin_database_password }}"
DatabaseName: "{{ app_name }}"
DomainName: nccopwatch.org
DomainNameAlternates: ""
AssetsCloudFrontDomain: files.nccopwatch.org
AssetsCloudFrontCertArn: arn:aws:acm:us-east-1:606178775542:certificate/379950bb-4b29-4308-8418-122674fe1076
AssetsUseCloudFront: "true"
tags:
Environment: "{{ app_name }}"
# cloudformation_stack:
# region: "{{ aws_region }}"
# stack_name: "{{ stack_name }}"
# template_bucket: "aws-web-stacks-{{ app_name }}"
# # Webstacks required variable
# template_local_path: '{{ playbook_dir + "/stack/eks-no-nat.yml" }}'
# create_changeset: true
# termination_protection: true

# template_parameters:
# PrimaryAZ: "{{ aws_region }}a"
# SecondaryAZ: "{{ aws_region }}b"
# DesiredScale: 2
# MaxScale: 4
# UseAES256Encryption: "true"
# CustomerManagedCmkArn: ""
# ContainerInstanceType: t3a.large
# ContainerVolumeSize: 40
# DatabaseAllocatedStorage: 100
# DatabaseClass: db.t3.large
# DatabaseEngineVersion: "12"
# DatabaseParameterGroupFamily: postgres12
# DatabaseMultiAZ: "false"
# DatabaseUser: "{{ app_name }}_admin"
# DatabasePassword: "{{ admin_database_password }}"
# DatabaseName: "{{ app_name }}"
# DomainName: nccopwatch.org
# DomainNameAlternates: ""
# AssetsCloudFrontDomain: files.nccopwatch.org
# AssetsCloudFrontCertArn: arn:aws:acm:us-east-1:606178775542:certificate/379950bb-4b29-4308-8418-122674fe1076
# AssetsUseCloudFront: "true"
# tags:
# Environment: "{{ app_name }}"

# Install Descheduler to attempt to spread out pods again after node failures
k8s_install_descheduler: yes
18 changes: 18 additions & 0 deletions deploy/group_vars/cdn.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
cloudformation_stack_state: present
cloudformation_stack_profile: '{{ aws_profile }}'
cloudformation_stack_region: '{{ aws_region }}'
cloudformation_stack_name: 'cdn-staging'
cloudformation_stack_template_bucket: 'aws-web-stacks-trafficstops'
cloudformation_stack_template_local_path: '{{ playbook_dir + "/stack/cloudfront.yml" }}'
cloudformation_stack_template_parameters:
AppCloudFrontCertArn: arn:aws:acm:us-east-1:606178775542:certificate/379950bb-4b29-4308-8418-122674fe1076
AppCloudFrontForwardedHeaders: "Host,Authorization"
DomainName: staging.nccopwatch.org
AppCloudFrontOriginDomainName: staging-origin.nccopwatch.org
AppCloudFrontRoleArn: cdn-staging
# Required for the CloudFront template
DomainNameAlternates: ""
CustomAppCertificateArn: ""
CertificateValidationMethod: "(none)"
cloudformation_stack_tags:
Environment: staging
226 changes: 226 additions & 0 deletions deploy/stack/cloudfront.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
# This Cloudformation stack template was generated by
# https://github.com/caktus/aws-web-stacks
# at 2024-10-03 10:18:42.500933
# with parameters:
# USE_CLOUDFRONT = on

Conditions:
AppCloudFrontCertArnCondition: !Not
- !Equals
- !Ref 'AppCloudFrontCertArn'
- ''
AppCloudFrontForwardedHeadersCondition: !Not
- !Equals
- !Join
- ''
- !Ref 'AppCloudFrontForwardedHeaders'
- ''
CertificateCondition: !Or
- !Not
- !Equals
- !Ref 'CustomAppCertificateArn'
- ''
- !Not
- !Equals
- !Ref 'CertificateValidationMethod'
- (none)
CustomAppCertArnCondition: !Not
- !Equals
- !Ref 'CustomAppCertificateArn'
- ''
NoAlternateDomains: !Equals
- !Join
- ''
- !Ref 'DomainNameAlternates'
- ''
StackCertificateCondition: !Not
- !Equals
- !Ref 'CertificateValidationMethod'
- (none)
UsEast1Condition: !Equals
- !Ref 'AWS::Region'
- us-east-1
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: Global
Parameters:
- DomainName
- DomainNameAlternates
- CertificateValidationMethod
- CustomAppCertificateArn
- Label:
default: Application Server
Parameters:
- AppCloudFrontOriginDomainName
- AppCloudFrontRoleArn
- AppCloudFrontProtocolPolicy
- AppCloudFrontForwardedHeaders
- AppCloudFrontCertArn
ParameterLabels:
AppCloudFrontCertArn:
default: CloudFront SSL Certificate ARN
AppCloudFrontForwardedHeaders:
default: CloudFront Forwarded Headers
AppCloudFrontOriginDomainName:
default: CloudFront Origin Domain Name
AppCloudFrontProtocolPolicy:
default: CloudFront Protocol Policy
AppCloudFrontRoleArn:
default: CloudFront Role ARN
CertificateValidationMethod:
default: Certificate Validation Method
CustomAppCertificateArn:
default: Custom App Certificate ARN
DomainName:
default: Domain Name
DomainNameAlternates:
default: Alternate Domain Names
Outputs:
AppCloudFrontDomainName:
Description: The app CDN domain name
Value: !GetAtt 'AppCloudFrontDistribution.DomainName'
Parameters:
AppCloudFrontCertArn:
Description: If your stack is NOT in the us-east-1 you must manually create an ACM certificate for your application domain in the us-east-1 region and provide its ARN here.
Type: String
AppCloudFrontForwardedHeaders:
Default: ''
Description: The headers that will be forwarded to the origin and used in the cache key. The 'Host' header is required for SSL on an Elastic Load Balancer, but it should NOT be passed to a Lambda Function
URL.
Type: CommaDelimitedList
AppCloudFrontOriginDomainName:
Default: ''
Description: Domain name of the origin server
Type: String
AppCloudFrontProtocolPolicy:
AllowedValues:
- redirect-to-https
- https-only
- allow-all
Default: redirect-to-https
Description: 'The protocols allowed by the application server''s CloudFront distribution. See: http://docs.aws.amazon.com/cloudfront/latest/APIReference/API_DefaultCacheBehavior.html'
Type: String
AppCloudFrontRoleArn:
Default: ''
Description: ARN of the role to add IAM permissions for invalidating this distribution
Type: String
CertificateValidationMethod:
AllowedValues:
- (none)
- DNS
- Email
Default: DNS
Description: >-
How to validate domain ownership for issuing an SSL certificate - highly recommend DNS. DNS and Email will pause stack creation until you do something to complete the validation. If omitted, an HTTPS
listener can be manually attached to the load balancer after stack creation.
Type: String
CustomAppCertificateArn:
Description: An existing ACM certificate ARN to be used by the application ELB. DNS and Email validation will not work with this option.
Type: String
DomainName:
Description: The fully-qualified domain name for the application.
Type: String
DomainNameAlternates:
Description: A comma-separated list of Alternate FQDNs to be included in the Subject Alternative Name extension of the SSL certificate.
Type: CommaDelimitedList
Resources:
AppCloudFrontDistribution:
Properties:
DistributionConfig:
Aliases: !Split
- ;
- !Join
- ''
- - !Ref 'DomainName'
- !If
- NoAlternateDomains
- ''
- ;
- !Join
- ;
- !Ref 'DomainNameAlternates'
DefaultCacheBehavior:
AllowedMethods:
- DELETE
- GET
- HEAD
- OPTIONS
- PATCH
- POST
- PUT
CachedMethods:
- HEAD
- GET
Compress: true
ForwardedValues:
Cookies:
Forward: all
Headers: !If
- AppCloudFrontForwardedHeadersCondition
- !Ref 'AppCloudFrontForwardedHeaders'
- !Ref 'AWS::NoValue'
QueryString: true
TargetOriginId: ApplicationServer
ViewerProtocolPolicy: !Ref 'AppCloudFrontProtocolPolicy'
Enabled: true
HttpVersion: http2
Origins:
- CustomOriginConfig:
OriginProtocolPolicy: https-only
DomainName: !Ref 'AppCloudFrontOriginDomainName'
Id: ApplicationServer
ViewerCertificate: !If
- UsEast1Condition
- AcmCertificateArn: !If
- CustomAppCertArnCondition
- !Ref 'CustomAppCertificateArn'
- !Ref 'Certificate'
MinimumProtocolVersion: TLSv1.2_2021
SslSupportMethod: sni-only
- !If
- AppCloudFrontCertArnCondition
- AcmCertificateArn: !Ref 'AppCloudFrontCertArn'
MinimumProtocolVersion: TLSv1.2_2021
SslSupportMethod: sni-only
- !Ref 'AWS::NoValue'
Tags:
- Key: aws-web-stacks:stack-name
Value: !Ref 'AWS::StackName'
Type: AWS::CloudFront::Distribution
AppCloudFrontInvalidationPolicy:
Properties:
PolicyDocument:
Statement:
- Action:
- cloudfront:GetDistribution
- cloudfront:GetDistributionConfig
- cloudfront:ListDistributions
- cloudfront:ListCloudFrontOriginAccessIdentities
- cloudfront:CreateInvalidation
- cloudfront:GetInvalidation
- cloudfront:ListInvalidations
Effect: Allow
Resource: '*'
PolicyName: AppCloudFrontInvalidationPolicy
Roles:
- !Ref 'AppCloudFrontRoleArn'
Type: AWS::IAM::Policy
Certificate:
Condition: StackCertificateCondition
Properties:
DomainName: !Ref 'DomainName'
DomainValidationOptions:
- DomainName: !Ref 'DomainName'
ValidationDomain: !Ref 'DomainName'
SubjectAlternativeNames: !If
- NoAlternateDomains
- !Ref 'AWS::NoValue'
- !Ref 'DomainNameAlternates'
Tags:
- Key: aws-web-stacks:stack-name
Value: !Ref 'AWS::StackName'
ValidationMethod: !Ref 'CertificateValidationMethod'
Type: AWS::CertificateManager::Certificate