Localstack (custom endpoint-url) support

[srgg]

Is your feature request related to a problem? Please describe.
Localstack requires to use custom endpoint url. Lack of aws endpoint configuration introduces a huge complexity for steampipe + localstack integration testing.

Describe the solution you'd like
It wil be great if custom endpoint url can be specified through plugin configuration mechanism.

[cbruno10]

Hey @srgg, thanks for opening this feature request!

I'm not familiar with LocalStack, but it seems like the AWS Go SDK supports a custom endpoint, and LocalStack even has a document on how to configure it. We configure and create our session mainly in this function:

steampipe-plugin-aws/aws/service.go

Lines 1976 to 2027 in f837d5c

	func getSessionWithMaxRetries(ctx context.Context, d *plugin.QueryData, region string, maxRetries int, minRetryDelay time.Duration) (*session.Session, error) {
	sessionCacheKey := fmt.Sprintf("session-%s", region)
	if cachedData, ok := d.ConnectionManager.Cache.Get(sessionCacheKey); ok {
	return cachedData.(*session.Session), nil
	}
	// If session was not in cache - create a session and save to cache
	// get aws config info
	awsConfig := GetConfig(d.Connection)
	// session default configuration
	sessionOptions := session.Options{
	SharedConfigState: session.SharedConfigEnable,
	Config: aws.Config{
	Region: &region,
	MaxRetries: aws.Int(maxRetries),
	Retryer: NewConnectionErrRetryer(maxRetries, minRetryDelay, ctx),
	},
	}
	if awsConfig.Profile != nil {
	sessionOptions.Profile = *awsConfig.Profile
	}
	if awsConfig.AccessKey != nil && awsConfig.SecretKey == nil {
	return nil, fmt.Errorf("Partial credentials found in connection config, missing: secret_key")
	} else if awsConfig.SecretKey != nil && awsConfig.AccessKey == nil {
	return nil, fmt.Errorf("Partial credentials found in connection config, missing: access_key")
	} else if awsConfig.AccessKey != nil && awsConfig.SecretKey != nil {
	sessionOptions.Config.Credentials = credentials.NewStaticCredentials(
	*awsConfig.AccessKey, *awsConfig.SecretKey, "",
	)
	if awsConfig.SessionToken != nil {
	sessionOptions.Config.Credentials = credentials.NewStaticCredentials(
	*awsConfig.AccessKey, *awsConfig.SecretKey, *awsConfig.SessionToken,
	)
	}
	}
	sess, err := session.NewSessionWithOptions(sessionOptions)
	if err != nil {
	plugin.Logger(ctx).Error("getSessionWithMaxRetries", "new_session_with_options", err)
	return nil, err
	}
	// save session in cache
	d.ConnectionManager.Cache.Set(sessionCacheKey, sess)
	return sess, nil
	}

In terms of configuring the AWS plugin, what would you expect an AWS plugin config connection to look like? For instance, would a connection per LocalStack environment with a different endpoint_url argument per connection be sufficient? Are there any other differences between LocalStack and a live AWS account that may require additional or updated config arguments, e.g., profile, regions, etc.?

[srgg]

@cbruno10 Thank you for the quick response.

For my limited Integration testing use case it should be enough to have just one generic endpoint-url being provided on per connection basis.

As far as I know there is no extra requirements from Localstack. But AWS provides different regional endpoints for the sake of compliance. AWS Go lang SDK (googled fro this) has ability to provide custom resolver, here is several useful links:

1. https://www.linkedin.com/pulse/how-setup-golang-aws-sdk-v2-localstack-local-phuc-hai-huynh/
2. https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/endpoints/

I have never used any of those, so I definetly not the right guy to decide how it can looks like.

Therefore what I can suggest/request for the simple case I have:

1. There can be a way to set up a default endpoint for all connections, that can be done by supporting environment variable AWS_ENDPOINT (not sure why that is not done by official SDK).
2. Nevertheless, should be a way to override default one provided via AWS_ENDPOINT by specifying endpoint-url for the particular connection.

[srgg]

@cbruno10 Thank you for pointing out exact place in the code, I've create a PR for the fix. One thing about that - I'm not sure how that change can be tested automatically, I've checked both options: AWS_ENDPOINT env variable and config with endpoit_url -- both seems to me working.

[cbruno10]

Thank you for working on and opening a PR so quickly @srgg, great stuff!

While looking around to see if AWS had an environment variable of their own today for the CLI or SDKs for setting an endpoint URL, I came across aws/aws-sdk#229, which contained a proposal from the AWS team on how to handle setting custom endpoints overall. Interestingly, they proposed adding config and environment variable support per service, with some alternative solutions for setting a global endpoint URL (among others).

We like to be as close and consistent with AWS as we can be, though sometimes we do need to deviate from their design principles due to the way plugins are designed. We're closely monitoring how they're thinking of implementing endpoint URLs beyond what they have today (the --endpoint-url CLI argument).

While using LocalStack, or other local AWS environments, do you ever have different endpoints per service, e.g., DynamoDB vs. S3 vs. EC2? Does LocalStack make all of these services available on the same endpoint? Also, do you ever use other local environments, like DynamoDB Local together with LocalStack?

A per service endpoint only design does not seem like it'd work well with LocalStack, assuming that all AWS services are available on the same endpoint in LocalStack.

Look forward to your thoughts on this!

[srgg]

@cbruno10 Thank you to pointing me out to the AWS ticket, I've made my suggestion to their proposal

Considering how long that epoch (lacking any support for endpoint_url) lasts , I'm personally, do not expect that they will implement/release this as high priority.

So the question: How do you guys want to move forward with that?

My interest is to get this merged in order to unblock integration testing over Localstack, I definitely, can make an adjustment by changing AWS_ENDPOINT to AWS_ENDPOINT_URL to fit to the proposed name.

Addressing your's questions:

• 'Yes', AFAIK Localstack provides all endpoints via the global link Article with DynamoDB usage examples, therefore for the sake of Localstack integration it was enough to introduce changes proposed in this PR Allow custom endpoint url configuration. Closes #1050 #1053.

• 'No', I haven't used anything except Localstack. I'm not aware about any other comprehensive solution that can run such a big set of AWS services locally. If you know, please point me out.

[cbruno10]

Thanks @srgg for adding your suggestion, I like the idea of having a global endpoint URL with the ability to override per service.

In regards to the AWS plugin, I think following a similar design would be helpful, so here are my proposed ideas on custom endpoints:

• Add a config argument, endpoint_url, which is a global custom endpoint and can be set like endpoint_url = "http://localhost:4566"
• Add an environment variable, AWS_ENDPOINT_URL, which aligns with the proposed AWS environment variable and our config arg.
• As required, add service specific config args and environment variables, similar to what's in the AWS proposal. I don't believe we need these yet, but we should watch when/how AWS implements these custom service endpoints and also prioritize adding them based on user requests.
• Order of precedence when evaluating which service endpoint URL to use:
  • Service specific config argument, e.g., dynamodb_endpoint_url, s3_endpoint_url
  • Service specific environment variable, e.g., AWS_DYNAMODB_ENDPOINT_URL, AWS_S3_ENDPOINT_URL
  • Global config argument, e.g., endpoint_url
  • Global environment variable, e.g., AWS_ENDPOINT_URL

There are plans to move this plugin to use AWS Go SDK v2 as well, and I believe that SDK should support the above proposal as well per Configuring Client Endpoints.

@srgg @johnsmyth - Any thoughts on the proposal above? Thanks!

[srgg]

@cbruno10 Proposed changes make a lot of sense to me. Moreover, I have updated PR and it covers the following parts of the proposal you have made:

• Add a config argument, endpoint_url, which is a global custom endpoint and can be set like endpoint_url = "http://localhost:4566"

• Add an environment variable, AWS_ENDPOINT_URL, which aligns with the proposed AWS environment variable and our config arg.
  ....

• Order of precedence when evaluating which service endpoint URL to use:

  • Global config argument, e.g., endpoint_url
  • Global environment variable, e.g., AWS_ENDPOINT_URL

Can you please review the PR one more time, and, hopefully, accept it?