Running Metastructure

Contents

• Running Metastructure
• CLI Overrides
• Config Expansion
  • Config Recursion
  • Format Expansion
  • CLI Params
  • SSO Reference
• Artifact Generation
• Authenticated Execution
• Config Updates
• Config Override Updates

Running Metastructure

Usage: metastructure [options] [command...]

Generate & manage infrastructure code.

Arguments:
  command                        Command to run within AWS authentication
                                 context.

Options:
  -w, --workspace <string>       Workspace name (required).
  -g, --generate                 Generate workspace from config.
  -u, --update-config            Update config from workspace output, conflicts
                                 with --update-override.
  -o, --update-override          Update config override from workspace output,
                                 conflicts with --update-config.
  -r, --assume-role <string>     Role to assume on target accounts (requires
                                 --aws-profile, conflicts with
                                 --permission-set).
  -p, --aws-profile <string>     AWS profile (requires --assume-role, conflicts
                                 with --permission-set).
  -s, --permission-set <string>  SSO permission set (conflicts with
                                 --assume-role & --aws-profile).
  -L, --local-state-on           Use local state (conflicts with -l).
  -l, --local-state-off          Use default state (conflicts with -L).
  -c, --config-path <string>     Config file path relative to CWD. Defaults to
                                 location specified in .metastructure.yml.
  -d, --debug                    Enable debug logging.
  -h, --help                     Display command help.

You can run Metastructure to do the following things. These may be specified in any combination, but will always run in this order:

• Generate artifacts (like Terraform code) in a selected Terraform workspace from your project config & Handlebars templates. Learn more...

• Create a valid AWS session (SSO or otherwise) and run a command (like terraform apply) against it. Learn more...

• Update your project config file from the output of your selected Terraform workspace. Learn more...

CLI Overrides

The following CLI arguments can be specified as defaults in your config file at workspaces.<workspace>.cli_defaults:

CLI Argument	Config Key	Config Type	Description
-r, --assume-role	assume_role	string | null	Role to assume on target accounts (requires aws_profile, conflicts with permission_set).
-p, --aws-profile	aws_profile	string | null	AWS profile (requires assume-role, conflicts with permission_set).
-s, --permission-set	permission_set	string | null	SSO permission set (conflicts with assume_role & aws_profile).
-L, --local-state-on -l, --local-state-off	use_local_state	boolean | null	Use local state.

Since the project config file will normally be persisted to your shared repository, these defaults apply to all developers working on the project.

If you wish to override these defaults locally, create a .gitignored local yaml file (e.g. workspace.local.yml) and add your overrides to the file root. For example:

assume_role: null
aws_profile: META-BOOTSTRAP
permission_set: terraform_admin

Then add the path to this file to your project config at workspaces.<workspace>.cli_defaults_path. These values will override the defaults in the config file.

Finally, you can override these values at the command line.

Some notes:

• To negate a CLI argument without overriding it, use null.

• CLI argument overrides are performed BEFORE CLI arguments are validated. After overrides, either permission_set or BOTH assume_role and aws_profile must be non-null.

• You can access resolved CLI arguments in your Handlebars templates at key cli_params.

Config Expansion

On loading, your project config file is expanded in several ways as described in the sections below.

When you run Metastructure, use the -d or --debug flag to see the expanded config object in your console!

Config Recursion

Your project config file is processed recursively as a Handlebars template, using itself as a data object. If your config file has this:

organization:
  tokens:
    audit_log: audit-logs
    namespace: metastructure-001
    owner: karmaniverous
accounts:
  dev:
    email: {{organization.tokens.owner}}+{{#if organization.tokens.namespace}}{{organization.tokens.namespace}}-{{/if}}dev@gmail.com

... it will be processed into this:

organization:
  tokens:
    audit_log: audit-logs
    namespace: metastructure-001
    owner: karmaniverous
accounts:
  dev:
    email: karmniverous+metastructure-001-dev@gmail.com

Format Expansion

For brevity, you can enter data into your project config file in a condensed form. For example, where the format wants an array of strings and you only have one string, you can just provide the string. Metastructure will convert it to an array before passing the config object to your Handlebars templates.

So this:

permission_sets:
  terraform_admin:
    policies: AdministratorAccess

... will become this:

permission_sets:
  terraform_admin:
    policies:
      - AdministratorAccess

There are also some special expansion cases that will be covered in the relevant config sections below.

CLI Params

As described in the CLI Overrides section above, CLI arguments are resolved and added to the config object at key cli_params.

SSO Reference

The groups, permission sets, and policies defined in the sso section of your project config file encode a complex set of relationships. Metastructure generates sso.reference keys to provide your templates with easy access to different facets of these relationships to facilitate the creation of the resources necessary to support SSO.

These include:

• account_permission_sets: A map of the permission sets assigned to each account across all groups. This is used by the session authentication engine to select an SSO profile based on the provided permission set.

• account_policies: A map of SSO policies assigned to each account via related permission sets. In the Metstructure Template repo, this is used by the bootstrap workspace SSO template to generate the aws_iam_policy resources necessary to support the SSO permission sets assigned to each account.

• group_account_permission_set_policies: A breakdown of the policies assigned to each permission set, in each account, in each SSO group. In the Metstructure Template repo, this is also used by the bootstrap workspace SSO template to generate the aws_ssoadmin_account_assignment resources that link these entities.

• policy_accounts: The inverse of account_policies, this is used by the bootstrap workspace SSO template to prevent a policy from being added to an SSO permission set until it has been created in its parent account.

Artifact Generation

Each workspace defined in your project config file contains an optional generators object. If present, each key of this object should specify a file path relative to the project root, and the associated value should specify the location of a Handlebars template, also relative to the project root.

When you run Metastructure with the -g or --generate flag, it will generate (or replace) the files specified in the generators keys by processing the associated Handlebars templates with the expanded project config data object for the workspace provided with the -w or --workspace flag.

To visualize this data object, use the -d or --debug flag when running Metastructure.

Neither templates nor destination files need be located in any specific directory, so it is perfectly reasonable to collect global templates in a common directory and reference them from multiple workspaces. This is the approach taken in the Metastructure Template Repo.

See the Handlebars Templates page for more information on how to write templates.

Authenticated Execution

There are two ways use Metastructure to generate AWS credentials, depending how your CLI overrides resolve.

If assume_role and aws_profile are populated, then Metastructure will attempt to authenticate with the indicated profile from your local AWS credentials file. These credentials will generally take the form of an AWS Access Key Id and an AWS Secret Access Key. Once authenticated, Metastructure will attempt to assume the indicated role. It is up to you to ensure that the credentials you provide have the necessary permissions to assume that role on all affected accounts.

If permission_set is populated, then there are a few more moving parts, which are best understood using the Metastructure Template repo as an example:

• The shared_config template generates an SSO credentials config file at the location specified in your project config file at workspaces.<workspace>.shared_config_path.

• Metastructure leverages this file to launch a browser window and initiate the SSO login process. You should log in as a user that has access to the indicated permission set.

• The backend and providers templates leverage the same shared config file to provide access to your Terraform state and account resources. GH-6

These steps assume code generation has taken place in the indicted workspace with a permission_set CLI argument in effect! Otherwise, the shared config file will not be populated, and the backend and providers files will be configured for key credential access rather than SSO.

So: if you are shifting from key credential access to SSO (which will happen as you bootstrap a new project), you MUST run Metastructure at least once with the -g or --generate flag in order to generate the required SSO configurations.

In practice, it is not unreasonable to run Metastructure with the -g or --generate flag all the time, unless you have a compelling reason not to. Any unexpected changes will show up in version control, so you can deal with them immediately.

In any case, when Metastructure is invoked with a command argument, that command will run within the context of the AWS session generated by the above process.

Consider this command:

metastructure -w bootstrap -g -u terraform apply

Assuming you already have either a permission_set or assume_role and aws_profile configured in cli_defaults, the above command below will:

• Generate the Terraform code for your bootstrap workspace.

• Establish an AWS session.

• Deploy your code to all accounts in your project.

• Update your project config with identifiers as described in Config Updates below.

Config Updates

When you run Metastructure with the -u or --update-config flag, Metastructure will merge the output of the current Terraform workspace with the contents of the config file.

The main purpose of this feature is to write identifiers of key resources (like accounts & OUs) back to the config file. Other templates can then use this information to decide whether to create new resources or import existing ones.

To use this feature:

• Create a template that generates a Terraform output containing the desired identifiers, whose structure matches that of the config file. It's ok to leave out irrelevant keys. Here's an example.

• Add the template to the relevant workspace's generators section in your project config file.

• Run Metastructure with the -u or --update-config flag.

Config Override Updates

If you are developing a new pattern or feature for the Metastructure Template Repo, you will need config updates to be available to future Metastructure executions, but will NOT want them written to the shared config file, since these values would have no place in a repository template.

In this case you should already be using a local Config Overrides file to avoid polluting the Template Repo with your test configurations.

If you run Metastructure with the -o or --update-override flag, instead of -u or --update-config, Metastructure will merge the output of the current Terraform workspace with the contents of your config overrides file rather than your project config file, thus preserving the integrity of the template.

See Template vs Project Mode for more info about managing project config while extending the Metastructure Template Repo.