-
-
Notifications
You must be signed in to change notification settings - Fork 0
Metastructure Config
Your Metastructure project config serves several purposes:
-
It records key global variables like Terraform versions & state configuration.
-
It defines key project entities, like AWS accounts & organizational units, applications, and environments, as well as the relationships between them.
-
It records key project entity identifiers as described in Config Updates.
-
It defines your SSO groups, permission sets, and policies, as well as their respective account relationships.
-
It defines your Terraform workspaces and the files to be generated via Handlebars template for each workspace.
-
It defines any other data elements that you want to use in your Handlebars templates.
Wherever possible, the config file format will accept data in a condensed form. See Format Expansion for more info.
Metastructure validates defined elements to ensure relational consistency. For example, it will validate that the account key assigned to an application environment actually exists. Feel free to create new elements of any complexity as you see fit, anywhere in the config file. Just know that validating the internal consistency of these elememts is up to you! See throwif in Handlebars Helpers for more info.
For you motivated developers, the Metastructure Config schema (a Zod schema) and the associated Typescript type, as well as all schema validations and expansions, are defined in this source file.
For everybody else, please explore the Metastructure Template Repo project config file. This is a working implementation and tracks current development, so it's a great example to follow!
The sections below briefly describe each section of the config file.
REMEMBER: You can add any data you like to your config file, anywhere you like, and use it in your Handlebars templates!
This section defines values that apply globally across your AWS Organization. It contains the following significant keys:
id is the AWS Organization ID. AWS Organizations must be created through the AWS Management Console, so you will need to enter this value manually, and your Organization will be imported into Terraform.
key_accounts is an object whose keys identify key functional roles within your organization and whose values identify the accounts that play them. For example, consider the following snippet:
organization:
key_accounts:
master: master
terraform_state: shared_servicesThis indicates that the the account with key master plays the master account role in your organization, and the account with key shared_services houses your Terraform state.
If your templates are suitably generic, this gives you the ability to assign key roles to accounts in your configuration, without having to make any template changes.
tokens is an object whose keys are used as tokens in your templates. For example, consider the following snippet:
organization:
tokens:
namespace: metastructure-001
accounts:
dev:
email: karmaniverous+{{#if organization.tokens.namespace}}{{organization.tokens.namespace}}-{{/if}}dev@gmail.comThanks to Config Recursion, the dev account email will resolve to karmaniverous+metastructure-001@gmail.com.
Organization tokens can be used throughout your configuration to provide a single source of truth for values that are used in multiple places.
This section identifies the AWS accounts in your organization. Each account is defined by a key, which is used to reference the account in other parts of the configuration, and a value, which is an object that defines the account's properties.
Each account object contains the following significant keys:
id is the AWS account id. If you add this value manually, Terraform will attempt to import your account and add it to your Organization. It's up to you to ensure the account and your permissions are properly configured to support this operation!
If your project follows the Metastructure Template Repo pattern, id will be populated when you perform a Config Update.
If id is populated and you add action: remove to the account object, when you run terraform apply the account will be removed from your Terraform state and your Organization. If you add action: destroy, the account will be removed from your Organization and destroyed. As always, this assumes you have already met the necessary conditions to perform these operations!
name is the name of the account. This is used in the AWS Management Console and in other places where human-readable account names are required.
email is the email address to be associated with the account. Metastructure validates that each account has a unique email, and remember that AWS requires every account to have a globally unique email address, which can't be reused on another account.
organizational_unit is optional, and is the key of the OU to which the account belongs. This key must match the key of an OU in the organizational_units section. To move an account to a new OU, simply update this value, regenerate your code & run terraform apply!
This section identifies the Organizational Units (OUs) in your organization. Each OU is defined by a key, which is used to reference the OU in other parts of the configuration, and a value, which is an object that defines the OU's properties.
Each OU object contains the following significant keys:
id & action work just like their account counterparts. See the accounts section above for details.
name is the name of the OU. This is used in the AWS Management Console and in other places where human-readable OU names are required.
parent is optional, and is the key of the parent OU. This key must match the key of another OU in the same section. To move an OU to a new parent, simply update this value, regenerate your code & run terraform apply!
This section defines the SSO groups, permission sets, and SSO-related policies in your organization.
The SSO section contains the significant keys laid out in the sections below.
start_url is the URL of the AWS SSO login page. This is used by the shared config template to generate the SSO credentials file.
users is an object whose properties define the SSO users in your organization. Each user is defined by a key, which is used as the User Name of the corresponding resource, and a value, which is an object that defines the user's properties.
Similar to accounts and organizational_units, User objects are "actionable". If you add an id key to a User, Metastructure will attempt to import the corresponding aws_identitystore_user resource. If an id already exists and you add action: remove or action: destroy, Terraform will attempt the corresponding operation.
When you run Metastructure with the -u, --update-config, -o, or --update-override flag, Metastructure will update your project config (or override) with ids of created resources.
User config values are validated by Metastructure, but the actionable & update behaviors described above are actually IMPLEMENTED in your templates! See the Metastructure Template Repo's bootstrap workspace SSO template for a working example.
User objects also contain SSO Group assignments, which are expanded as described in Format Expansion.
Here's an example of this section:
sso:
users:
# Creates a user for the organization owner. Note the use of Config
# Recursion in the user key! For more info see
# https://github.com/karmaniverous/metastructure/wiki/Running-Metastructure#config-recursion
'{{ organization.tokens.owner }}':
# Format Expansion at work here.
groups: terraform_admin
# These keys are required to create the aws_identitystore_user resource
# but are not strictly required by Metastructure. Add whatever keys your
# templates need!
email: '{{organization.tokens.owner}}@{{organization.tokens.domain}}'
display_name: Test User
name:
given_name: Test
family_name: User
'john-doe':
groups:
- terraform_admin
- some_other_group
email: 'john.doe@{{organization.tokens.domain}}'
display_name: John Doe
# Format Expansion at work here.
name:
given_name: John
family_name: DoeNote that while you certainly CAN create & manage your SSO users with Metastructure, most significant organizations SHOULDN'T! Instead, they should use an external identity provider (IdP) like Active Directory or Google Workspace.
In its current form, Metastructure will support that use case just fine... just add any necessary keys to your project config & consume them with your templates. What we do NOT yet have is a working example of this in the Metastructure Template Repo.
So stay tuned! Meanwhile, we'll use the native AWS IdP to demonstrate SSO at Hello, Enterprise!
groups is an object whose properties define the SSO groups in your organization. Each group is defined by a key, which is used to reference the group in other parts of the configuration, and a value, which is an object that defines the group's properties.
There's some special expansion behavior here that is best illustrated by example:
sso:
groups:
# This is the key used to reference the SSO Group in other parts of the configuration.
terraform_admin:
# These are the SSO Group name & description that appear in the IAM Identity Center console.
name: TerraformAdmin
description: Terraform administrators can create & manage all resources in all accounts.
# These are the keys of the SSO Permission Sets (see below) assigned to the SSO Group. In this example, the terraform_admin permission set will be assigned to all accounts.
account_permission_sets: terraform_admin
audit_reader:
name: AuditReader
# In this example, both of the indicated permission sets will be assigned to all accounts.
account_permission_sets:
- audit_log_reader
- app_log_reader
non_prod_audit_reader:
name: NonProdAuditReader
# In this example, both of the indicated permission sets will be assigned to the designated accounts.
account_permission_sets:
dev:
- audit_log_reader
- app_log_reader
test:
- audit_log_reader
- app_log_readerpermission_sets is an object whose properties define the SSO permission sets in your organization. Each permission set is defined by a key, which is used to reference the permission set in other parts of the configuration, and a value, which is an object that defines the permission set's properties.
The point of this section is to connect permission sets with assigned policies in the relevant accounts. It's up to you and your templates to actually CREATE those policies using the expanded config object. See the Metastructure Template Repo's bootstrap workspace SSO template for a working example.
Here's an annotated example of this section:
sso:
permission_sets:
# This is the key used to reference the SSO Permission Set in other parts of the configuration.
terraform_admin:
# These are the SSO Permission Set name & description that appear in the IAM Identity Center console.
name: TerraformAdmin
description: Permits creation & management of all resources.
# These are the keys of the AWS IAM Policies (see below) assigned to the SSO Permission Set. IMPORTANT: AWS Managed Policies (like AdministratorAccess below) should be referenced by name.
policies:
- AdministratorAccess
- sso_terraform_state_writerpolicies is an object whose keys represent the user-defined IAM Policies that will support SSO in each relevant account, and whose value represents the name of that policy. Only user-defined policies should be referenced here. AWS Managed Policies should be referenced by name in the permission_sets section.
This section is generated by Metastructure. It exposes facets of the relationships between SSO groups, permission sets, and policies in a way that can be directly consumed by your templates to create the resources necessary to support SSO.
You should not add content to this section manually, as it will be overwritten when you run Metastructure and generate your config object.
Here's an example of this section as generated by Metastructure:
{
"sso": {
"reference": {
"account_permission_sets": {
"dev": ["terraform_admin", "terraform_deployment"],
"log_archive": ["terraform_admin", "terraform_deployment"],
"master": ["terraform_admin", "terraform_deployment"],
"prod": ["terraform_admin", "terraform_deployment"],
"test": ["terraform_admin", "terraform_deployment"],
"shared_services": ["terraform_admin", "terraform_deployment"]
},
"account_policies": {
"dev": ["sso_terraform_state_writer", "unprotected_resource_writer"],
"log_archive": [
"sso_terraform_state_writer",
"unprotected_resource_writer"
],
"master": ["sso_terraform_state_writer", "unprotected_resource_writer"],
"prod": ["sso_terraform_state_writer", "unprotected_resource_writer"],
"test": ["sso_terraform_state_writer", "unprotected_resource_writer"],
"shared_services": [
"sso_terraform_state_writer",
"unprotected_resource_writer"
]
},
"group_account_permission_set_policies": {
"terraform_admin": {
"dev": { "terraform_admin": ["sso_terraform_state_writer"] },
"log_archive": { "terraform_admin": ["sso_terraform_state_writer"] },
"master": { "terraform_admin": ["sso_terraform_state_writer"] },
"prod": { "terraform_admin": ["sso_terraform_state_writer"] },
"test": { "terraform_admin": ["sso_terraform_state_writer"] },
"shared_services": {
"terraform_admin": ["sso_terraform_state_writer"]
}
},
"terraform_deployment": {
"dev": { "terraform_deployment": ["sso_terraform_state_writer"] },
"log_archive": {
"terraform_deployment": ["sso_terraform_state_writer"]
},
"master": { "terraform_deployment": ["sso_terraform_state_writer"] },
"prod": { "terraform_deployment": ["sso_terraform_state_writer"] },
"test": { "terraform_deployment": ["sso_terraform_state_writer"] },
"shared_services": {
"terraform_deployment": ["sso_terraform_state_writer"]
}
}
},
"permission_set_accounts": {
"terraform_admin": [
"dev",
"log_archive",
"master",
"prod",
"test",
"shared_services"
],
"terraform_deployment": [
"dev",
"log_archive",
"master",
"prod",
"test",
"shared_services"
]
},
"policy_accounts": {
"sso_terraform_state_writer": [
"dev",
"log_archive",
"master",
"prod",
"test",
"shared_services"
],
"unprotected_resource_writer": [
"dev",
"log_archive",
"master",
"prod",
"test",
"shared_services"
]
}
}
}
}This section defines the applications that your organization uses, the environments each application runs in, and the accounts that host those environments.
The applications section is a little lean at the moment: Metastructure evaluates it for referential integrity, but it is not yet consumed by any templates in the Metastructure Template Repo.
More to come!
This section defines global values that apply specifically to Terraform and Terraform state.
Mostly its keys are consumed by templates, notably to generate backend & provider resources.
The one important exception to this is the paths key, which is used to define the path or paths (if you use an array) to your Terraform state files. Metastructure invokes Terraform to format the files in these paths after it generates code from your templates.
This is where the magic happens!
Each key in the workspaces object is the name of a Terraform workspace. Each value is an object that defines the workspace's properties. These objects have the following significant properties:
cli_defaults sets global CLI defaults for the workspace. See CLI Overrides above for more information.
cli_defaults_path is the path (relative to the project root) to a local YAML file that contains CLI overrides for the workspace. See CLI Overrides above for more information.
path identifies the path (relative to the project root) to the current working directory to be used when running Terraform for this workspace.
shared_config_path is the path (relative to the project root) to the shared config file that will be used by backend & provider files for SSO authentication. GH-6
generators is an object whose keys identify the files to be generated by the workspace and whose values identify the Handlebars templates to be used to generate them. Both sets of paths should be expressed relative to the project root.
partials Defines workspace-specific Handlebars partials that can also override global partials. See Handlebars Partials for more info.
Defines global Handlebars partials that can be used in any workspace. See Handlebars Partials for more info.
TODO
Sometimes you will want to set a Metastructure config value in your local development environment without persisting that change back to the remote repo.
For example, the Metastructure Template Repo project config file presents a generic set of organization tokens, which must be overridden with specific values to support testing of the Template Repo.
To support this use case, the config_override_path key in your project config is configured to point to a gitignored override file. See metastructure.local.yml.template for instructions on creting this file. To use a different override file name, just change the config_override_path to match the new file path.
If this override file exists, Metastructure will deep-merge its contents with your project config file before processing config expansions. If the override file does not exist, even if it is specified at config_override_path, Metastructure will proceed without error.
See Template vs Project Mode for more info about managing project config while extending the Metastructure Template Repo.
Clone the Metastructure Template Repo to get started!
Built for you with ❤️ on Bali! Find more great tools & templates on my GitHub Profile.