Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support custom S3 endpoint URIs #209

Closed
mikeseese opened this issue Jun 1, 2022 · 4 comments · Fixed by #210
Closed

Support custom S3 endpoint URIs #209

mikeseese opened this issue Jun 1, 2022 · 4 comments · Fixed by #210

Comments

@mikeseese
Copy link

mikeseese commented Jun 1, 2022

To be able to use non-AWS S3 buckets (Cloudflare R2, DigitalOcean Spaces, etc), you need to specify a custom endpoint URI. Unfortunately, the AWS SDK doesn't provide an environment or config file variable to override this value (aws/aws-cli#4454); it only supports a CLI parameter or using the direct SDK variable). So for longtail to support these custom endpoints, the SDK references would need to be changed

Using the docs here, here, here, and here, I believe that

client := s3.NewFromConfig(cfg)
can be modified to look something like (please excuse my lack of golang experience):

client := s3.NewFromConfig(cfg, func(o *s3.Options) {
	if ConfiguredBaseEndpointUri != nil {
		o.EndpointResolver := s3.EndpointResolverFromURL(ConfiguredBaseEndpointUri)
	}
})

or

if ConfiguredBaseEndpointUri != nil {
	cfg.EndpointResolver := s3.EndpointResolverFromURL(ConfiguredBaseEndpointUri)
}
client := s3.NewFromConfig(cfg)

where ConfiguredBaseEndpointUri could be some new CLI flag (i.e. --s3-base-endpoint).

It's also very possible that it's as simple as the below (I've seen another example say aws.Config has Endpoint, but the official docs say otherwise)

if ConfiguredBaseEndpointUri != nil {
	cfg.Endpoint := ConfiguredBaseEndpointUri
}
client := s3.NewFromConfig(cfg)

Once the endpoint can be configured, users can configure the region and credentials using the ~/.aws/config and ~/.aws/credentials files mentioned in longtail's readme or use AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables (which get read in during the LoadDefaultConfig call, so there's no reason to make any changes to support those variables.

I would attempt to implement this myself and make a PR, but my golang experience is measured in weeks and I would likely just be going down a rabbithole for simple issues

@mikeseese mikeseese changed the title Support custom S3 endpoint URI Support custom S3 endpoint URIs Jun 1, 2022
@DanEngelbrecht
Copy link
Owner

Thanks for the info, I'll take a peek once time permits, hopefully within a week or two.
I don't have any of the non-AWS S3 buckets so testing might be an issue, would you be willing to test some stuff out once I have something @seesemichaelj ?

@mikeseese
Copy link
Author

Absolutely!

@DanEngelbrecht
Copy link
Owner

@seesemichaelj Please try out https://github.com/DanEngelbrecht/golongtail/releases/tag/v0.3.4-pre14 and see how the new option feels and works out.,

@mikeseese
Copy link
Author

mikeseese commented Jun 7, 2022

@DanEngelbrecht I can confirm that v0.3.4-pre14 works with DigitalOcean S3 Spaces using the --s3-endpoint-resolver-uri https://sfo3.digitaloceanspaces.com as the endpoint configuration (and specifying the region, key, and secret variables via AWS config files and environment variables). Awesome, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants