From 4ede23f5c20e54a7dacb04e137452f80a9269d13 Mon Sep 17 00:00:00 2001 From: Annu Singh Date: Tue, 5 Mar 2024 04:08:58 +0530 Subject: [PATCH 1/7] docs: amore details about orchestration --- docs/cli/orchestration/index.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/cli/orchestration/index.md b/docs/cli/orchestration/index.md index c6002e63a..ab44f3a17 100644 --- a/docs/cli/orchestration/index.md +++ b/docs/cli/orchestration/index.md @@ -32,3 +32,32 @@ The default order of execution can be altered in a stack's configuration. For de You can use the [list --run-order](../cmdline/list.md) command to understand the order of execution of your stacks. ::: + +## Ways to Run Commands + +In Terramate, you have two main ways to run commands: + +1. **Terramate Run Command:** + Use this for running single, straightforward commands in your setup. It's great for quickly doing specific tasks across your infrastructure. + + Run a command in all stacks: + ```bash + terramate run terraform init + ``` + +2. **Terramate Scripts ([workflows](scripts.md)):** + These are like playbooks for more complicated jobs. With Terramate Scripts, you can set up and run sequences of commands, automating complex processes. They're perfect for managing bigger tasks and making sure everything runs smoothly. + +These two ways to orchestrate commands in Terramate give you everything you need to manage your commands, from simple tasks to bigger, more complicated workflows. Whether you're running one-off commands or managing a whole series of actions, Terramate has you covered. + + +## Change Detection Integration + +Terramate offers seamless integration with [change detection](../change-detection/index.md), allowing users to optimize command orchestration by focusing only on the modified stacks. This section highlights how users can leverage the `terramate run` and `terramate script run` commands in conjunction with the `--changed` flag to facilitate this process. + +By pairing the `--changed` flag with the `terramate run` command, users can instruct Terramate to execute commands exclusively on stacks that have undergone modifications since the last execution. Similarly, when using the `terramate script run` command, appending the `--changed` flag enables users to trigger the execution of Terramate Scripts solely on stacks with detected changes. + +This feature makes it easier for users to handle resources effectively, especially when there are changes in the setup of their infrastructure. + +## Sequential and Parallel Command Orchestration +Terramate CLI supports both sequential and parallel execution of commands to orchestrate stacks, thus accommodating both dependent and independent stacks. The CLI utilizes a [fork-join model](https://en.wikipedia.org/wiki/Fork%E2%80%93join_model) to execute the sequential parts (dependent stacks requiring specific order) and parallel parts (independent stacks that can be executed in any order). By leveraging this approach, Terramate ensures efficient execution while maintaining accuracy and consistency across deployments. \ No newline at end of file From 6591232e69661d666713c4ab99698390ed372bc0 Mon Sep 17 00:00:00 2001 From: Annu Singh Date: Tue, 5 Mar 2024 06:21:31 +0530 Subject: [PATCH 2/7] docs: sequential & parallel execution --- docs/cli/orchestration/index.md | 45 ++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/docs/cli/orchestration/index.md b/docs/cli/orchestration/index.md index ab44f3a17..b7e7b5872 100644 --- a/docs/cli/orchestration/index.md +++ b/docs/cli/orchestration/index.md @@ -60,4 +60,47 @@ By pairing the `--changed` flag with the `terramate run` command, users can inst This feature makes it easier for users to handle resources effectively, especially when there are changes in the setup of their infrastructure. ## Sequential and Parallel Command Orchestration -Terramate CLI supports both sequential and parallel execution of commands to orchestrate stacks, thus accommodating both dependent and independent stacks. The CLI utilizes a [fork-join model](https://en.wikipedia.org/wiki/Fork%E2%80%93join_model) to execute the sequential parts (dependent stacks requiring specific order) and parallel parts (independent stacks that can be executed in any order). By leveraging this approach, Terramate ensures efficient execution while maintaining accuracy and consistency across deployments. \ No newline at end of file +Terramate CLI supports both sequential and parallel execution of commands to orchestrate stacks, thus accommodating both dependent and independent stacks. The CLI utilizes a [fork-join model](https://en.wikipedia.org/wiki/Fork%E2%80%93join_model) to execute the sequential parts (dependent stacks requiring specific order) and parallel parts (independent stacks that can be executed in any order). By leveraging this approach, Terramate ensures efficient execution while maintaining accuracy and consistency across deployments. + +### Sequential +Stacks that have a dependency on other stacks need to be run sequentially. +Terramate cli runs [nested stacks](../stacks/nesting.md) in sequence as per the order of execution defined in the stack's configuration. + +In Terramate, by default, commands are executed sequentially. When adhering to the default order of execution for a stack hierarchy as illustrated below: +```sh +/vpc + /network_acl + /internet_gateway + /subnet + /network_interface + /ec2 + /route_table + /security_group +``` +The sequence of execution follows this pattern: + +1. `vpc` +2. `network_acl` +3. `internet_gateway` +4. `subnet` + - `network_interface` + - `ec2` +5. `route_table` +6. `security_group` + +As an illustration, to execute a command sequentially across all stacks within a particular directory: +```bash +terramate run --chdir stacks/vpc -- terraform init +``` +### Parallel + +Terramate facilitates parallel execution, enabling independent stacks to run in parallel, thereby offering significant time savings, particularly during commands like `terramate run terraform init`. Despite the parallel nature of execution, Terramate ensures that the order of execution is still respected. + +This approach notably diminishes build time consumption for deployments and drift detection to a bare minimum, while also reducing waiting time for users when executing commands across stacks. Moreover, with Terramate Cloud, users can conveniently access logs of all executed stacks in the correct order, further enhancing visibility and monitoring capabilities during the execution process. + +To initiate parallel execution, users can utilize the `--parallel=N` flag, where N represents the number of parallel processes desired. This allows users to tailor the level of parallelism according to their specific requirements. +For example: +```bash +terramate run --parallel=5 terraform init +``` +This command runs `terraform init` in parallel across all stacks while maintaining the specified order. \ No newline at end of file From 49e48c49fffb2e18680fe305ea00cd405751cd6e Mon Sep 17 00:00:00 2001 From: Annu Singh Date: Thu, 7 Mar 2024 00:59:04 +0530 Subject: [PATCH 3/7] Update docs/cli/orchestration/index.md Co-authored-by: i4k --- docs/cli/orchestration/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cli/orchestration/index.md b/docs/cli/orchestration/index.md index b7e7b5872..47cba54a1 100644 --- a/docs/cli/orchestration/index.md +++ b/docs/cli/orchestration/index.md @@ -98,7 +98,7 @@ Terramate facilitates parallel execution, enabling independent stacks to run in This approach notably diminishes build time consumption for deployments and drift detection to a bare minimum, while also reducing waiting time for users when executing commands across stacks. Moreover, with Terramate Cloud, users can conveniently access logs of all executed stacks in the correct order, further enhancing visibility and monitoring capabilities during the execution process. -To initiate parallel execution, users can utilize the `--parallel=N` flag, where N represents the number of parallel processes desired. This allows users to tailor the level of parallelism according to their specific requirements. +To initiate parallel execution, users can utilize the `--parallel N` flag, where N represents the number of parallel processes desired. This allows users to tailor the level of parallelism according to their specific requirements. For example: ```bash terramate run --parallel=5 terraform init From 591a1e9a79bea788ab55aafd1f91590e64d5997b Mon Sep 17 00:00:00 2001 From: Annu Singh Date: Thu, 7 Mar 2024 00:59:56 +0530 Subject: [PATCH 4/7] Update docs/cli/orchestration/index.md Co-authored-by: i4k --- docs/cli/orchestration/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cli/orchestration/index.md b/docs/cli/orchestration/index.md index 47cba54a1..da1cba34e 100644 --- a/docs/cli/orchestration/index.md +++ b/docs/cli/orchestration/index.md @@ -94,7 +94,7 @@ terramate run --chdir stacks/vpc -- terraform init ``` ### Parallel -Terramate facilitates parallel execution, enabling independent stacks to run in parallel, thereby offering significant time savings, particularly during commands like `terramate run terraform init`. Despite the parallel nature of execution, Terramate ensures that the order of execution is still respected. +Terramate facilitates parallel execution, enabling independent stacks to run in parallel, thereby offering significant time savings, particularly during commands like `terraform init`. Despite the parallel nature of execution, Terramate ensures that the order of execution is still respected. This approach notably diminishes build time consumption for deployments and drift detection to a bare minimum, while also reducing waiting time for users when executing commands across stacks. Moreover, with Terramate Cloud, users can conveniently access logs of all executed stacks in the correct order, further enhancing visibility and monitoring capabilities during the execution process. From ffbcab019e2d5fa8da425ed13036b7b4d9c77caa Mon Sep 17 00:00:00 2001 From: Annu Singh Date: Tue, 19 Mar 2024 15:18:45 +0530 Subject: [PATCH 5/7] correct execution sequence --- docs/cli/orchestration/index.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/cli/orchestration/index.md b/docs/cli/orchestration/index.md index da1cba34e..cbee59b1d 100644 --- a/docs/cli/orchestration/index.md +++ b/docs/cli/orchestration/index.md @@ -77,17 +77,20 @@ In Terramate, by default, commands are executed sequentially. When adhering to t /route_table /security_group ``` -The sequence of execution follows this pattern: - -1. `vpc` -2. `network_acl` -3. `internet_gateway` -4. `subnet` - - `network_interface` - - `ec2` -5. `route_table` -6. `security_group` +When executed sequentially and respecting the nested layout, the execution sequence follows this pattern (in alphabetical order): +```sh +terramate list --run-order + +vpc +vpc/internet_gateway +vpc/network_acl +vpc/route_table +vpc/security_group +vpc/subnet +vpc/subnet/ec2 +vpc +``` As an illustration, to execute a command sequentially across all stacks within a particular directory: ```bash terramate run --chdir stacks/vpc -- terraform init From 34bfce2bbb168fcc358747dd87d01845c6257574 Mon Sep 17 00:00:00 2001 From: Tiago Natel Date: Thu, 21 Mar 2024 15:26:55 +0000 Subject: [PATCH 6/7] tests: ensure tests do not use GITHUB_TOKEN exported. Signed-off-by: Tiago Natel --- cmd/terramate/e2etests/internal/runner/runner.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmd/terramate/e2etests/internal/runner/runner.go b/cmd/terramate/e2etests/internal/runner/runner.go index c3c4e7690..e7ae50d5f 100644 --- a/cmd/terramate/e2etests/internal/runner/runner.go +++ b/cmd/terramate/e2etests/internal/runner/runner.go @@ -74,7 +74,7 @@ func NewCLI(t *testing.T, chdir string, env ...string) CLI { } if len(env) == 0 { // by default, it's assumed human mode - env = RemoveEnv(os.Environ(), "CI", "GITHUB_ACTIONS") + env = RemoveEnv(os.Environ(), "CI", "GITHUB_ACTIONS", "GITHUB_TOKEN") } env = append(env, "CHECKPOINT_DISABLE=1") // custom cliconfig file From 0f32a985c51f2a7d968341864900f59306ced84e Mon Sep 17 00:00:00 2001 From: Tiago Natel Date: Thu, 21 Mar 2024 19:15:12 +0000 Subject: [PATCH 7/7] chore: bump dev version. Signed-off-by: Tiago Natel --- VERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/VERSION b/VERSION index e54f09939..fb0b75423 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.6.0-rc1 +0.6.0-dev