Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retryablehttp Client Option #619

Merged
merged 11 commits into from
Aug 3, 2023
Merged

Add retryablehttp Client Option #619

merged 11 commits into from
Aug 3, 2023

Conversation

danaelhe
Copy link
Member

@danaelhe danaelhe commented Jul 13, 2023

Users can opt to use the httpclient from the retryablehttp package when creating a godo client to enable automatic retries and exponential backoff.

Users can enable using the retryablehttp httpclient by passing in the SetRetryMax() option when creating a client using New(). Users can then configure the retryWaitMax, and retryWaitMin values.

This first iteration uses the retryablehttp's DefaultRetryPolicy and DefaultBackOff. By not explicitly setting them, they are the defaults as shown here.

For the test, I passed in retryMax=2 and retryWaitMax=6.0 seconds and retryWaitMin=6.0s:

=== RUN   TestRetryableClient_DefaultRetryPolicy
2023/07/13 11:47:20 [DEBUG] GET http://127.0.0.1:53811/foo
2023/07/13 11:47:20 [DEBUG] GET http://127.0.0.1:53811/foo (status: 500): retrying in 6s (2 left)
2023/07/13 11:47:26 [DEBUG] GET http://127.0.0.1:53811/foo (status: 500): retrying in 6s (1 left)
--- PASS: TestRetryableClient_DefaultRetryPolicy (12.00s)
PASS
ok      github.com/digitalocean/godo    12.706s

As you can see, it retries 2 times and waits 6 seconds inbetween retries. I'm not sure how to explicitly check the test captured the number of retries and time between in this test- so open to suggestions.

@danaelhe danaelhe requested a review from a team July 13, 2023 16:00
godo.go Outdated Show resolved Hide resolved
retryableClient.RetryWaitMax = time.Duration(c.RetryConfig.RetryWaitMin * float64(time.Second))

// if timeout is set, it is maintained before overwriting client with StandardClient()
retryableClient.HTTPClient.Timeout = c.client.Timeout
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find that some opensource packages will also set a timeout, so decided to maintain it as well.

@danaelhe danaelhe changed the base branch from main to feature/godo_with_retryable July 18, 2023 19:48
@danaelhe danaelhe requested review from andrewsomething and a team July 18, 2023 19:55
@danaelhe danaelhe changed the base branch from feature/godo_with_retryable to main July 21, 2023 16:22
@danaelhe danaelhe changed the base branch from main to feature/godo_with_retryable July 26, 2023 18:20
@danaelhe danaelhe changed the base branch from feature/godo_with_retryable to main July 26, 2023 18:21
@danaelhe danaelhe changed the base branch from main to feature/retryablehttp_option July 26, 2023 20:51
@danaelhe danaelhe changed the base branch from feature/burstlimit_config to main August 2, 2023 13:23
// RetryConfig sets the values used for enabling retries and backoffs for
// requests that fail with 429 or 500-level response codes using the go-retryablehttp client.
// RetryConfig.RetryMax must be configured to enable this behavior. RetryConfig.RetryWaitMin and
// RetryConfig.RetryWaitMax are optional, with the default values being 1.0 and 30.0, respectively.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RetryWaitMax and RetryWaitMin do have default values: https://github.com/hashicorp/go-retryablehttp/blob/main/client.go#L51

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those defaults look good based on the testing I did over the course of looking at this branch, so I'm happy with using those 👍

retryableClient.RetryWaitMin = time.Duration(*c.RetryConfig.RetryWaitMin * float64(time.Second))
}
if c.RetryConfig.RetryWaitMax != nil {
retryableClient.RetryWaitMax = time.Duration(*c.RetryConfig.RetryWaitMax * float64(time.Second))
Copy link
Member Author

@danaelhe danaelhe Aug 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An effort to avoid an unset RetryWaitMax and RetryWaitMin with a value of 0.0 overriding the retryablehttp's default value of 30 seconds and 1 second, respectively.

type RetryConfig struct {
RetryMax int
RetryWaitMin *float64 // Minimum time to wait
RetryWaitMax *float64 // Maximum time to wait
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed RetryWaitMin and RetryWaitMax to pointers to differentiate between an unset variable and a variable set to 0.0.

Copy link
Member

@bentranter bentranter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall! A bunch of nitpicks (since once we merge and release this we're pretty much stuck with it), but the implementation/approach is solid.

go.mod Outdated Show resolved Hide resolved
// RetryConfig sets the values used for enabling retries and backoffs for
// requests that fail with 429 or 500-level response codes using the go-retryablehttp client.
// RetryConfig.RetryMax must be configured to enable this behavior. RetryConfig.RetryWaitMin and
// RetryConfig.RetryWaitMax are optional, with the default values being 1.0 and 30.0, respectively.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those defaults look good based on the testing I did over the course of looking at this branch, so I'm happy with using those 👍

godo.go Show resolved Hide resolved
godo.go Outdated Show resolved Hide resolved
godo_test.go Outdated Show resolved Hide resolved
Copy link
Member

@bentranter bentranter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm good with this 👍

@danaelhe danaelhe merged commit bc798b0 into main Aug 3, 2023
7 checks passed
@danaelhe danaelhe deleted the godo_retryablehttp branch August 3, 2023 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants