Migrate static.crates.io to Fastly

[jdno]

Context

As laid out in our (soon-to-be-released) documentation on how we use CDNs, we want to move static.crates.io and static.rust-lang.org to Fastly to reduce the cost of outbound traffic on AWS.

static.crates.io has been chosen since it is the distribution with the 2nd largest traffic volume. But it does not use cache invalidations or custom routing, so no implementation changes are required. This makes it the ideal candidate to start with. The distribution will be migrated as-is, without any changes in functionality or scope.

Tasks

Implement Compute@Edge function

• Create a Compute@Edge function
  • Limit HTTP methods to GET and HEAD
  • Fail-over between S3 buckets based on HTTP response
• Document how to build and deploy the function

Set up staging service

• Create fastly_service_compute resource with the Compute@Edge function
• Configure environment using fastly_service_dictionary_items
• Set up Let's Encrypt so that Fastly can manage certificates for static.staging.crates.io
• Create fastly_tls_subscription resource to enable TLS
• Create aws_route53_record resources for ACME DNS challenge
• Create fastly_tls_subscription_validation resource to generate certificate
• Add static.staging.crates.io as an alternate name to TLS subscription
• Test that Fastly can serve traffic for static.staging.crates.io
• Explicitly set TTL for static content

Migrate static.staging.crates.io domain

• Create cloudfront-static.staging.crates.io record
• Create Traffic Policy to split traffic between CloudFront and Fastly
• Test that traffic is distributed as expected
• Test Fastly service using public domain and confirm that responses are identical to CloudFront

Set up production service

• Deploy Fastly service for static.crates.io
• Add static.crates.io as an alternate name to TLS subscription
• Test that Fastly can serve traffic for static.crates.io
• Explicitly set TTL for static content

Migrate static.crates.io

• Create cloudfront-static.crates.io record
• Create Traffic Policy to split traffic between CloudFront and Fastly
• Test that traffic is distributed as expected
• Test Fastly service using public domain and confirm that responses are identical to CloudFront
• Announce rollout in a blog post to the wider community
• Change weight incrementally with enough time in between to ensure that there are no issues

[jdno]

I've updated the plan based on some lessons learned in the draft pull request for this change. In particular, the following things have changed:

• Our S3 buckets are publicly available and don't require authentication. While we might still want to implement private S3 buckets, it's not required for the initial roll-out.
• We'll be using the new Compute@Edge platform, since the traditional VCL infrastructure does not support failing over between origins based on HTTP responses.
• Compute@Edge forces traffic to use TLS by itself, which means we don't need to set up a custom condition that does the same.

[jdno]

The configuration for Fastly seems stable enough that we can start testing to gather some data. Only the alternate name in the Fastly certificate still has to be deployed to production so that we can test static.crates.io.

Once that is done, we can overwrite the DNS resolution for static.crates.io locally to send traffic to Fastly instead of CloudFront. If everything works as expected, this will be completely transparent and tools should just work™. This should give a good first indication of the performance and reliability of the service.

The final piece left to do is to set up weighted records so that we can split traffic between CloudFront and Fastly on the DNS level. For staging, we probably want to run 100% of traffic through Fastly. For production, we want to have a gradual rollout to observe performance, reliability, and cost.

My suggestion is that we take this week (January 9th) to test internally. If no issues come up, let's start pushing a little bit more traffic each day (e.g. 1%) to Fastly beginning next week (January 16th).

[jdno]

We've been slowly rolling out Fastly for static.crates.io over the past two weeks. We'll continue to increase the ratio of traffic, but I'll go ahead and close this issue.