Remove region from stack_master.yml

[patrobinson]

This is a Proposal for the direction of 2.0 of StackMaster. The proposal started here and given there wasn't a great deal of discussion on this I thought I'd throw it up.

I often find the duplication of stacks in stack_master.yml to be tedious and unnecessary. We often duplicate stack configuration many times, especially when we are building multiple environments and multi-region deployments.

Essentially we are duplicating the region parameter twice, once in stack_master.yml and once on the command line. We can remove it from the config file, but it does have some draw backs:

• The need for a migration "tool" to upgrade existing configurations to 2.0. I'll add this as part of a future PR
• We can no longer have per region per stack configurations for notification_arns, s3_buckets and files. The configuration must be for an entire region or an entire stack. For 2.0 we also plan on supporting per account settings called "environments", so this will likely also become a place to configure these settings (and probably per region per environment).
• It breaks existing CI workflows we use where we just run stack_master validate to validate all our stacks. validate, status and diff now must be passed a region.

Alternatively we could move these configurations into the parameter files, but that doesn't feel like where it should go. Other proposals for where this configuration might live are welcome.

Ultimately I think this proposal moves us closer to a better experience for StackMaster users, by making it simpler to manage multiple stacks deployed to multiple regions and environments with little heavy lifting. Eventually we want stack_master.yml file to look something like this:

---
environments:
  production: 012345678910
  staging: 109876543210
bundles:
  my-web-app:
    vpc:
      template: vpc.rb
    web:
      template: asg.rb
    db:
      template: rds.rb
    redis:
      template: redis.rb
stacks:
  my-bastion:
    template: bastion.rb

Where I can stack_master apply us-east-1 my-web-app and have it come up cleanly.

This PR is to be merged into the version-2.0 branch, which will act as our develop branch until 2.0 is considered ready for a release.

[patrobinson]

Actually this will break the way we use validate in more ways. Because we don't give our CI environment access to the S3 buckets that stacks are uploaded to and if any stack has to be uploaded to s3 all stacks will be uploaded to s3 and our workflow will be broken.

But we don't really need to validate all stacks just all templates. So I will raise a new PR in 1.0 to make validate do just that.

[grassdog]

I think something along these lines makes sense, though I feel like region is just a characteristic of another unexpressed concept. Ignoring the migration path for a bit, I feel like stacks (or perhaps bundles from your example above) should be applied into different targets (or deployments or some better name, not sure about environment). These could be staging, sandbox, production, production-failover, my-throw-away-test-deployment etc...

Each target would require a region and optionally have other parameters which override things in the stack itself. Something like the following perhaps:

# Stack
- name
- templates
- base parameters

# Each stack has one or more `targets` which has the following

- name
- region
- parameters which override those base ones defined in the stack (optional)

Thoughts?

[mipearson]

I very much like the "targets" idea, @grassdog

[patrobinson]

I think this helps solve another problem, which is how can we ensure you are in the right AWS account for the right "target", by making the targets static (in config), not ephemeral (on the command line).

[rbayerl]

FWIW this will very much break how my team uses Stack Master. For numerous regions we have the same stacks deployed in multiple accounts and regions, but they have different stack names. It's fine if this is the direction Stack Master is heading, but it will likely be too much work for us to refactor everything (we have hundreds of projects with multiple stack_master.yml files). It would be more helpful to me and my team to merge #221 as a bug fix for a v1 release (since this seems like a bug and there is nothing saying only a single alias is supported per region).

Take this comment with a grain of salt...we don't really need any other features and if we want more parameter resolvers we can patch them in to our tooling manually. Also we have mitigated the aforementioned bug by just using more stack_master.yml files (although this is more confusing and complicated for CI/CD). So ultimately we will probably just pin stack_master to ~> 1.

[arecker]

Going to also throw my hat in there along with @rbayerl !

Ultimately I think this proposal moves us closer to a better experience for StackMaster users, by making it simpler to manage multiple stacks deployed to multiple regions and environments with little heavy lifting.

This would result in having less infrastructure represented in code. In our previous system, to make a change to a stack you had to know (1) the account, (2) the region, and (3) the stack name. You can imagine how over the moon we were when we found a solution that captured both (2) and (3) in code!

Whatever you decide, just wanted to call out how important it is to us for the end result to be more (or the same) amount of infrastructure captured in code - not less.

[mipearson]

Taking @grassdog's target idea and trying to resolve @arecker & @rbayerl's issues above, how about something like:

---
targets:
  production-us:
    account_id: 012345678910
    region: us-east-1
  production-eu:
    account_id: 012345678910
    region: eu-west-1
stacks:
  us-bastion:
    target: production-us
    template: bastion.rb
  eu-bastion:
    target: production-eu
    template: bastion.rb
    # stack name override (optional, eg supporting existing stacks)
    stack_name: bastion-eu-old

Not sure how this would work w/ bundles, and not sure how it'd work for common parameters between templates that are essentially the same between different targets.

[mipearson]

Another potential idea (although I'm a bit worried about indirection / abstraction):

targets:
  production-us:
    account_id: 012345678910
    region: us-east-1
  production-eu:
    account_id: 012345678910
    region: eu-west-1
stacks:
  bastion:
    template: bastion.rb
    # some config can go here ?
deployments:
  production-us:
    us-bastion:
      stack: bastion
  production-eu:
    eu-bastion:
      stack: bastion
      # override, as above
      stack_name: bastion-eu-old 

[patrobinson]

Thanks a lot for your feedback @rbayerl and @arecker. Can you provide your feedback on @grassdog and @mipearson 's proposals too?

I like @grassdog's proposal a lot and I feel this is a better solution than this PR, but it's going to have to require some more thought on how we model the concept of "stacks" and "targets" or "deployments".

[rbayerl]

I do find the concept intriguing. It would be nice to have all of this metadata in the config file itself. The idea of targets would help us to decrease the number of config files we have currently too. The only thing I would be worried about is how Stack Master would find credentials - we've seen some pretty good examples of how not to do it 🙂. That's more of an implementation detail though.

I think @arecker said it much more succinctly when he said we don't want to lose any of the existing infrastructure as code.

[patrobinson]

@rbayerl I generally defer finding credentials to the SDK provider. That has a well defined, well known, lookup order

Tools such as aws-vault help with managing multiple accounts and access keys. They can be used in conjunction with StackMaster without us needing to explicitly support it, it just works.

I'm open to supporting specifying what profile to use to lookup credentials using the CLI configuration, as this is pretty simple and well defined integration.

I don't have any other proposals for how we manage credentials but I'm open to them, please create a Github issue if you want to propose something else.