Making Redshift cluster type a first class config, or removing it alltogether?

[Fleid]

Hey all,

I'm looking at #171 and #138 and even #94, and I'm left wondering what we should do with cluster type in terms of config.

Currently there are mostly 3 cluster types available as a Redshift user, independently of dbt:

• DC2/DS2 nodes, older node types
• RA3, more flexible querying and scaling
• Serverless, most recent, with usage based billing (and lots of other goodies)

The current default behaviour in dbt is to assume DC2/DS2, and enable "RA3 mode" by using ra3_node in the profile. See #171 for why we may actually not really need it anymore.

Now when trying to enable IAM authentication on Serverless, in #138, we're realizing that we may need a flag to differentiate the authentication logic for Serverless. In #139 we see a serverless profile config appearing.

Now I'm not a big fan of having multiple parameters to describe a single property (is_ra3:true|false + is_serverless:true|false) when what we need to know is actually cluster_type: ra3|serverless|dc2.... Refactoring toward a single config is a first option here.

Another option would be to try to abstract that away, and build logic to handle the authentication across cluster types, transparently to the user. One way could be to parse the hostname and check if serverless's in there - but it's not perfect. And what about other features, specific to a cluster type, that we may need to surface anyway?

There doesn't seem to be an obvious answer here, so I was wondering if anyone had strong feelings / opinions on the topic.

Thanks :)

[adam-bloom]

@Fleid when IAM authentication is involved, so are AWS API calls. Can any of this be discerned from the redshift describeclusters API response? I'm not sure that it would fully solve this, but it'd be nice to not need to configure something that dbt can discover.

[mp24-git]

@adam-bloom The DescribeClusters API only returns information on provisioned Redshift clusters. It does not return anything for serverless, for which a completely separate API is implemented (unfortunately).

[mp24-git]

I think that the cluster_type is actually the best option, given that:

• serverless "workgroups" have all the features of RA3 nodes. Code doing RA3-specific stuff (see [CT-1061] Remove ra3_node profile config? #171) would need to handle serverless as well
• IAM authentication requires some flag for serverless to use the correct redshift or redshift-serverless API. Even the Redshift Python driver has a separate parameter is_serverless

It would of course be possible to determine whether the given cluster is serverless or not by attempting the DescribeClusters API and the GetWorkgroup API (for serverless) and see whether any of it returns a result. However, this would lead to IAM issues when using tightly restricted IAM roles.

[Fleid]

Sorry for the delayed answer.

@mp24-git It looks like the is_serverless flag is necessary, whether we stay with our current authentication method or we eventually switch to the Redshift Python library:

Connecting to a Redshift Serverless Endpoint
Authentication methods discussed below are supported for Redshift serverless endpoints. If connecting using a network load balancer (NLB) or Redshift-managed VPC endpoint please set is_serverless=True and specify workgroup and serverless account id information using serverless_acct_id and serverless_work_group, respectively.

From what I understand, we can't reliably get that info from an API, so we need the user to make that call.

Now it seems like we don't need to differentiate between provisioned cluster node types anymore (between DC2 and RA3 via the RA3 flag), but instead between Redshift SKUs/Services (provisioned vs serverless via new flag) - not event sure we can call them that.

Two options ahead of us:

• Add a new is_serverless flag, in addition to RA3_node
  • Independently, we may want to retire RA3_node
  • The short term plan feels good, we only differentiate between cluster types.
  • But if AWS either introduce multiple flavors of serverless, or differentiate provisioned SKUs harder, we will end in a bad place where we have flags that can contradict themselves
• Add a new sku flag and retiring RA3_node
  • I don't like that name, but I dislike it less than cluster_type since serverless doesn't have clusters
  • Which for now would default to provisioned, but can be set to serverless
  • That we can easily extend later via additional values (serverlessV2,XYZ...) if/when needs be