Implement ctile #7
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds context.Context to the various client calls made by the proxy to enable deadlining, cancellation, and other handy things. This doesn't choose a top-level request timeout, but allows for one to be set in another PR.
Using the previous context code, apply some timeouts to ensure the HTTP server stays up under traffic. This weaves a default 4 second per-request timeout through the handler. This timeout covers how long it takes the HTTP handler to return data from S3 (if it exists) or to discover the tile isn't in S3, fetch the tile data from S3 the backend, and write it before returning. This is configurable. This also sets the basic HTTP server timeouts that are required for healthy Go HTTP servers. Those aren't configurable. A good summary of Go's HTTP server timeouts and why they're needed can be found at https://blog.cloudflare.com/exposing-go-on-the-internet/ The `TimeoutHandler` sets the timeout on the `http.Request`'s context, so we can use `r.Context()` in the handler to propagate the deadline to the S3 and HTTP clients. The values for the timeouts were chosen arbitrarily but from values that I understand to be safe for public consumption. The `full-request-timeout` default is the one most likely to need a change depending on how far from S3 this server is hosted (or if the work to write to S3 is moved to a background goroutine or whatever). Future work: Given that we're using `http.Server` directly now, it might be time to break out the handler func as a top-level function but I leave that for a possible follow-up.
Qualify the S3 key by the backend URL and the tile size. Also, keep track of the backend URL in the `tile` object.
Also add -s3-prefix override (defaults to -backend)
This moves the HTTP code out of the main func and into its own type called `tileCachingHandler`. It's a simple refactoring that adds some fields to the new struct to keep the pattern of adding flags in main.
aarongable
reviewed
Aug 31, 2023
Trillian only applies the align_getentries tweak for "maximally sized" requests, so CTile using a tile size smaller than max_get_entries is fine. Co-authored-by: Preston Locke <me@prestonlocke.net>
aarongable
approved these changes
Aug 31, 2023
.github/workflows/test.yml
Outdated
| @@ -4,7 +4,7 @@ on: | |||
| push: | |||
| branches: [ main ] | |||
| pull_request: | |||
| branches: [ main ] | |||
| branches: '*' | |||
There was a problem hiding this comment.
You want ** so that it matches branches with / in their name
Suggested change
| branches: '*' | |
| branches: '**' |
aarongable
approved these changes
Aug 31, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
nota bene: land this change using the "rebase" method, so that these initial commits are preserved