[tf] Add support for meta roles in HA deployments

[MichaelThamm]

Currently we deploy worker role juju_application resources with a scale of 1 unit (

observability/terraform/modules/loki-ha/variables.tf

Line 43 in 739d2de

default = 1

). So when a user deploys with terraform apply the default behaviour is that all microservice worker roles are deployed (without meta roles).

We want to be able to support meta roles and deployment types (monolithic vs. microservice) with TF but this introduces some complexities. Here are some options:

Option 1: Default to 0 units and require the user to define each role (or meta role).

e.g.

module "loki_read" {
  count = var.read_units > 0 ? 1 : 0  // THIS LINE IS IMPORTANT so the app does not get created with 0 units
  config = {
    role-read = true
  }
  units = var.read_units
}

and when you include the Loki module (coordinator + workers) in something like COS, you need to specify each worker role to deploy (since the default is 0):

module "loki" {
  source = "<path_to_loki_module>"
  read_units = 1
  write_units = 1
  backend_units = 1
}

The downside of this approach is that it adds more conditional logic within the TF files since we have no guarantee if an app exists since we conditionally create them. Therefore need some try functions like try(module.loki.read_app) in outputs.tf or other files where we want to access the app.

Option 2: Make our TF deployments opinionated.

Have 1 TF module for monolithic and 1 for microservice. Although this introduces code duplication, this would remove the complexities (conditional logic) from our TF files. Additionally, the user then does not need to specify each unit when including the Loki module:

this

module "loki" {
  source = "<path_to_loki_module>"
  read_units = 1
  write_units = 1
  backend_units = 1
}

becomes

module "loki" {
  source = "<path_to_loki_monolithic_module>"
}

or

module "loki" {
  source = "<path_to_loki_microservice_module>"
}