terraform-aws-cloud

This repository contains opinionated Terraform modules used to deploy and configure an AWS EKS cluster for the StreamNative Platform. It is currently underpinned by the terraform-aws-eks module.

The working result is a Kubernetes cluster sized to your specifications, bootstrapped with StreamNative's Platform configuration, ready to receive a deployment of Apache Pulsar.

For more information on StreamNative Platform, head on over to our official documentation.

Prerequisites

The Terraform command line tool is required and must be installed. It's what we're using to manage the creation of a Kubernetes cluster and its bootstrap configuration, along with the necessary cloud provider infrastructure.

We use Helm for deploying the StreamNative Platform charts on the cluster, and while not necessary, it's recommended to have it installed for debugging purposes.

Your caller identity must also have the necessary AWS IAM permissions to create and work with EC2 (EKS, VPCs, etc.) and Route53.

Other Recommendations

• aws command-line tool
• aws-iam-authenticator command line tool

Networking

EKS has multiple modes of network configuration for how you access the EKS cluster endpoint, as well as how the node groups communicate with the EKS control plane.

This Terraform module supports the following:

• Public (EKS) / Private (Node Groups): The EKS cluster API server is accessible from the internet, and node groups use a private VPC endpoint to communicate with the cluster's controle plane (default configuration)
• Public (EKS) / Public (Node Groups): The EKS cluster API server is accessible from the internet, and node groups use a public EKS endpoint to communicate with the cluster's control plane. This mode can be enabled by setting the input enable_node_group_private_networking = false in the module.

Note: Currently we do not support fully private EKS clusters with this module (i.e. all network traffic remains internal to the AWS VPC)

For your VPC configuration we require sets of public and private subnets (minimum of two each, one per AWS AZ). Both groups of subnets must have an outbound configuration to the internet. We also recommend using a seperate VPC reserved for the EKS cluster, with a minimum CIDR block per subnet of /24.

A Terraform sub-module is available that manages the VPC configuration to our specifications. It can be used in composition to the root module in this repo (see this example).

For more information on how EKS networking can be configured, refer to the following AWS guides:

• Networking in EKS
• Amazon EKS cluster endpoint access control
• De-mystifying cluster networking for Amazon EKS worker nodes

Getting Started

A bare minimum configuration to execute the module:

data "aws_eks_cluster" "cluster" {
  name = module.eks_cluster.eks_cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks_cluster.eks_cluster_name
}

provider "aws" {
  region = var.region
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  insecure               = false
}

variable "region" {
  default = "us-east-1"
}

module "sn_cluster" {
  source = "streamnative/cloud/aws"

  cluster_name                   = "sn-cluster-${var.region}"
  cluster_version                = "1.21"
  hosted_zone_id                 = "Z04554535IN8Z31SKDVQ2" # Change this to your hosted zone ID
  node_pool_max_size             = 3

  ## Note: EKS requires two subnets, each in their own availability zone
  public_subnet_ids  = ["subnet-abcde012", "subnet-bcde012a"]
  private_subnet_ids = ["subnet-vwxyz123", "subnet-efgh242a"]
  region             = var.region
  vpc_id             = "vpc-1234556abcdef"
}

In the example main.tf above, a StreamNative Platform EKS cluster is created using Kubernetes version 1.21.

By default, the cluster will come provisioned with 8 node groups (reference node topology¹), six of which have a desired capacity set to 0, and only the "xlarge" node group has a default desired capacity of 1. All

Creating a StreamNative Platform EKS Cluster

When deploying StreamNative Platform, there are additional resources to be created alongside (and inside!) the EKS cluster:

• StreamNative operators for Pulsar
• Vault Configuration & Resources

We have made this easy by creating additional Terraform modules that can be included alongside your EKS module composition. Consider adding the following to the example main.tf file above:

#######
### This module installs the necessary operators for StreamNative Platform
### See: https://registry.terraform.io/modules/streamnative/charts/helm/latest
#######
module "sn_bootstrap" {
  source = "streamnative/charts/helm"

  enable_function_mesh_operator = true
  enable_vault_operator         = true
  enable_pulsar_operator        = true

  depends_on = [
    module.sn_cluster,
  ]
}

To apply the configuration initialize the Terraform module in the directory containing your own version of the main.tf from the examples above:

terraform init

Validate and apply the configuration:

terraform apply

Deploy a StreamNative Platform Workload (an Apache Pulsar Cluster)

We use a Helm chart to deploy StreamNative Platform on the receiving Kubernetes cluster. Refer to our official documentation for more info.

Note: Since this module manages all of the Kubernetes addon dependencies required by StreamNative Platform, it is not necessary to perform all of the steps outlined in the Helm chart's README.. Please reach out to your customer representative if you have questions.

Requirements

Name	Version
terraform	>= 1.3.2
aws	5.75.0
kubernetes	2.32.0

Providers

Name	Version
aws	5.75.0

Modules

Name	Source	Version
eks	terraform-aws-modules/eks/aws	20.29.0
eks_auth	terraform-aws-modules/eks/aws//modules/aws-auth	20.29.0
vpc_tags	./modules/eks-vpc-tags	n/a

Resources

Name	Type
aws_ec2_tag.cluster_security_group	resource
aws_iam_role.cluster	resource
aws_iam_role.ng	resource
aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy	resource
aws_iam_role_policy_attachment.cluster_AmazonEKSServicePolicy	resource
aws_iam_role_policy_attachment.cluster_AmazonEKSVPCResourceControllerPolicy	resource
aws_iam_role_policy_attachment.ng_AmazonEKSServicePolicy	resource
aws_iam_role_policy_attachment.ng_AmazonEKSVPCResourceControllerPolicy	resource
aws_iam_role_policy_attachment.ng_AmazonEKSWorkerNodePolicy	resource
aws_caller_identity.current	data source
aws_iam_policy_document.cluster_assume_role_policy	data source
aws_iam_policy_document.ng_assume_role_policy	data source
aws_kms_key.ebs_default	data source
aws_partition.current	data source
aws_subnet.private_subnets	data source
aws_subnet.public_subnets	data source

Inputs

Name	Description	Type	Default	Required
add_vpc_tags	Adds tags to VPC resources necessary for ingress resources within EKS to perform auto-discovery of subnets. Defaults to "true". Note that this may cause resource cycling (delete and recreate) if you are using Terraform to manage your VPC resources without having a lifecycle { ignore_changes = [ tags ] } block defined within them, since the VPC resources will want to manage the tags themselves and remove the ones added by this module.	bool	true	no
additional_tags	Additional tags to be added to the resources created by this module.	map(any)	{}	no
allowed_public_cidrs	List of CIDR blocks that are allowed to access the EKS cluster's public endpoint. Defaults to "0.0.0.0/0" (any).	list(string)	[ "0.0.0.0/0" ]	no
bootstrap_self_managed_addons	Indicates whether or not to bootstrap self-managed addons after the cluster has been created	bool	null	no
cluster_enabled_log_types	A list of the desired control plane logging to enable. For more information, see Amazon EKS Control Plane Logging documentation (https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html).	list(string)	[ "api", "audit", "authenticator", "controllerManager", "scheduler" ]	no
cluster_encryption_config	Configuration block with encryption configuration for the cluster. To disable secret encryption, set this value to {}	any	{}	no
cluster_iam	Cluster IAM settings	any	null	no
cluster_name	The name of your EKS cluster and associated resources. Must be 16 characters or less.	string	""	no
cluster_networking	Cluster Networking settings	any	null	no
cluster_security_group_additional_rules	Additional rules to add to the cluster security group. Set source_node_security_group = true inside rules to set the node_security_group as source.	any	{}	no
cluster_security_group_id	The ID of an existing security group to use for the EKS cluster. If not provided, a new security group will be created.	string	""	no
cluster_service_ipv4_cidr	The CIDR block to assign Kubernetes service IP addresses from. If you don't specify a block, Kubernetes assigns addresses from either the 10.100.0.0/16 or 172.20.0.0/16 CIDR blocks	string	null	no
cluster_version	The version of Kubernetes to be installed.	string	"1.20"	no
create_cluster_security_group	Whether to create a new security group for the EKS cluster. If set to false, you must provide an existing security group via the cluster_security_group_id variable.	bool	true	no
create_node_security_group	Whether to create a new security group for the EKS nodes. If set to false, you must provide an existing security group via the node_security_group_id variable.	bool	true	no
disable_public_eks_endpoint	Whether to disable public access to the EKS control plane endpoint. If set to "true", additional configuration is required in order for the cluster to function properly, such as AWS PrivateLink for EC2, ECR, and S3, along with a VPN to access the EKS control plane. It is recommended to keep this setting to "false" unless you are familiar with this type of configuration.	bool	false	no
disk_encryption_kms_key_arn	The KMS Key ARN to use for EBS disk encryption. If not set, the default EBS encryption key will be used.	string	""	no
enable_bootstrap	deprecated	bool	false	no
enable_cilium	deprecated	bool	false	no
enable_istio	deprecated	bool	false	no
enable_node_pool_monitoring	Enable CloudWatch monitoring for the default pool(s).	bool	false	no
enable_nodes_use_public_subnet	When set to true, the node groups will use public subnet rather private subnet, and the public subnet must enable auto-assing public ip so that nodes can have public ip to access internet. Default is false.	bool	false	no
enable_resource_creation	deprecated	bool	true	no
enable_sncloud_control_plane_access	Whether to enable access to the EKS control plane endpoint. If set to "false", additional configuration is required in order for the cluster to function properly, such as AWS PrivateLink for EC2, ECR, and S3, along with a VPN to access the EKS control plane. It is recommended to keep this setting to "true" unless you are familiar with this type of configuration.	bool	true	no
enable_v3_node_groups	Enable v3 node groups, which uses a single ASG and all other node groups enabled elsewhere	bool	false	no
enable_v3_node_migration	Enable v3 node and v2 node groups at the same time. Intended for use with migration to v3 nodes.	bool	false	no
enable_v3_node_taints	When v3 node groups are enabled, use the node taints. Defaults to true	bool	true	no
enable_vpc_cni_prefix_delegation	Whether set ENABLE_PREFIX_DELEGATION for vpc-cni addon	bool	true	no
iam_path	An IAM Path to be used for all IAM resources created by this module. Changing this from the default will cause issues with StreamNative's Vendor access, if applicable.	string	"/StreamNative/"	no
manage_aws_auth_configmap	Whether to manage the aws_auth configmap	bool	true	no
map_additional_iam_roles	A list of IAM role bindings to add to the aws-auth ConfigMap.	list(object({ rolearn = string username = string groups = list(string) }))	[]	no
node_groups	Map of EKS managed node group definitions to create	any	null	no
node_pool_ami_id	The AMI ID to use for the EKS cluster nodes. Defaults to the latest EKS Optimized AMI provided by AWS.	string	""	no
node_pool_azs	A list of availability zones to use for the EKS node group. If not set, the module will use the same availability zones with the cluster.	list(string)	[]	no
node_pool_desired_size	Desired number of worker nodes in the node pool.	number	0	no
node_pool_disk_iops	The amount of provisioned IOPS for the worker node root EBS volume.	number	3000	no
node_pool_disk_size	Disk size in GiB for worker nodes in the node pool. Defaults to 50.	number	100	no
node_pool_disk_type	Disk type for worker nodes in the node pool. Defaults to gp3.	string	"gp3"	no
node_pool_ebs_optimized	If true, the launched EC2 instance(s) will be EBS-optimized. Specify this if using a custom AMI with pre-user data.	bool	true	no
node_pool_instance_types	Set of instance types associated with the EKS Node Groups. Defaults to ["m6i.large", "m6i.xlarge", "m6i.2xlarge", "m6i.4xlarge", "m6i.8xlarge"], which will create empty node groups of each instance type to account for any workload configurable from StreamNative Cloud.	list(string)	[ "m6i.large", "m6i.xlarge", "m6i.2xlarge", "m6i.4xlarge", "m6i.8xlarge" ]	no
node_pool_labels	A map of kubernetes labels to add to the node pool.	map(string)	{}	no
node_pool_max_size	The maximum size of the node pool Autoscaling group.	number	n/a	yes
node_pool_min_size	The minimum size of the node pool AutoScaling group.	number	0	no
node_pool_pre_userdata	The user data to apply to the worker nodes in the node pool. This is applied before the bootstrap.sh script.	string	""	no
node_pool_tags	A map of tags to add to the node groups and supporting resources.	map(string)	{}	no
node_pool_taints	A list of taints in map format to apply to the node pool.	any	{}	no
node_security_group_additional_rules	Additional ingress rules to add to the node security group. Set source_cluster_security_group = true inside rules to set the cluster_security_group as source	any	{}	no
node_security_group_id	An ID of an existing security group to use for the EKS node groups. If not specified, a new security group will be created.	string	""	no
permissions_boundary_arn	If required, provide the ARN of the IAM permissions boundary to use for restricting StreamNative's vendor access.	string	null	no
private_subnet_ids	The ids of existing private subnets.	list(string)	[]	no
public_subnet_ids	The ids of existing public subnets.	list(string)	[]	no
region	The AWS region.	string	null	no
use_runtime_policy	Legacy variable, will be deprecated in future versions. The preference of this module is to have the parent EKS module create and manage the IAM role. However some older configurations may have had the cluster IAM role managed seperately, and this variable allows for backwards compatibility.	bool	false	no
v3_node_group_core_instance_type	The instance to use for the core node group	string	"m6i.large"	no
vpc_id	The ID of the AWS VPC to use.	string	""	no

Outputs

Name	Description
eks	All outputs of module.eks for provide convenient approach to access child module's outputs.
eks_cluster_arn	The ARN for the EKS cluster created by this module
eks_cluster_certificate_authority_data	Base64 encoded certificate data required to communicate with the cluster
eks_cluster_endpoint	The endpoint for the EKS cluster created by this module
eks_cluster_identity_oidc_issuer_arn	The ARN for the OIDC issuer created by this module
eks_cluster_identity_oidc_issuer_string	A formatted string containing the prefix for the OIDC issuer created by this module. Same as "cluster_oidc_issuer_url", but with "https://" stripped from the name. This output is typically used in other StreamNative modules that request the "oidc_issuer" input.
eks_cluster_identity_oidc_issuer_url	The URL for the OIDC issuer created by this module
eks_cluster_name	The name of the EKS cluster created by this module
eks_cluster_platform_version	The platform version for the EKS cluster created by this module
eks_cluster_primary_security_group_id	The id of the primary security group created by the EKS service itself, not by this module. This is labeled "Cluster Security Group" in the EKS console.
eks_cluster_secondary_security_group_id	The id of the secondary security group created by this module. This is labled "Additional Security Groups" in the EKS console.
eks_node_group_iam_role_arn	The IAM Role ARN used by the Worker configuration
eks_node_group_security_group_id	Security group ID attached to the EKS node groups
eks_node_groups	Map of all attributes of the EKS node groups created by this module
inuse_azs	The availability zones in which the EKS nodes is deployed

Footnotes

1. When running Apache Pulsar in Kubernetes, we make use of EBS backed Kubernetes Persistent Volume Claims (PVC). EBS volumes themselves are zonal, which means an EC2 instance can only mount a volume that exists in its same AWS Availability Zone. For this reason we have added node group "zone affinity" functionality into our module, where an EKS node group is created per AWS Availability Zone. This is controlled by the number of subnets you pass to the EKS module, creating one node group per subnet. In addition, we also create node groups based on instance classes, which allows us to perform more fine tuned control around scheduling and resource utilization. To illustrate, if a cluster is being created across 3 availability zones and the default 4 instance classes are being used, then 12 total node groups will be created, all except the nodes belonging to the xlarge (which has a default capicty of 1 for initial scheduling of workloads) group will remain empty until a corresponding Pulsar or addon workload is deployed. ↩