-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
TER-405 add doc and help for tf-module harvest (#163)
document these commands, and update command help text: - harvest resources - harvest modules - harvest mappings Also, update table of contents
- Loading branch information
Kanak Singhal
authored
Dec 12, 2023
1 parent
5622be1
commit 5dc63c5
Showing
12 changed files
with
341 additions
and
51 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| # Terraform Mappings Harvester | ||
|
|
||
| The Terraform Mappings Harvester is a specialized tool designed to parse Terraform code and its modules to identify and scrape resource attribute mappings. | ||
|
|
||
| ## Working | ||
|
|
||
| ```mermaid | ||
| flowchart TD | ||
| TC["Terraform HCL code"] --"terraform init"--> T[".terraform/modules"] | ||
| T --> MC["Inner Module code\n.terraform/modules/module_name/*.tf"] | ||
| MC --> MH["Mappings Harvester"] | ||
| MH --> DB["Database\ntf_resource_attribute_mappings"] | ||
| ``` | ||
|
|
||
| The Terraform Mappings Harvester operates through a series of steps to identify and extract resource attribute mappings from Terraform code. The process, as visualized in the above Mermaid diagram, involves the following key stages: | ||
|
|
||
| 1. **Terraform HCL Code Processing**: | ||
| - The process begins with the Terraform HCL (HashiCorp Configuration Language) code. | ||
| - The user runs `terraform init` on their Terraform code. This command is crucial for initializing a Terraform working directory, which includes downloading and setting up the necessary modules. | ||
|
|
||
| 2. **Extraction of Inner Module Code**: | ||
| - The initialization process creates a `.terraform/modules` directory. | ||
| - Within this directory, the Harvester focuses on the inner module code, which is located in `.terraform/modules/module_name/*.tf`. These files contain the actual Terraform code for each module used in the project. | ||
| - This step is critical as it involves parsing the Terraform files to identify the resource attribute mappings. | ||
|
|
||
| 3. **Mappings Harvester Processing**: | ||
| - The Mappings Harvester then processes the extracted module code. | ||
| - It systematically scans through the `.tf` files, identifying and extracting mappings between input and output resource attributes. For example, it can detect a mapping like `aws_security_group.vpc_id = aws_vpc.id`, indicating that the `vpc_id` attribute of `aws_security_group` maps to the `id` attribute of `aws_vpc` by analyzing at a code block like this: | ||
|
|
||
| ```tf | ||
| resource "aws_security_group" "dbsg" { | ||
| name = "db" | ||
| description = "security group for db" | ||
| vpc_id = aws_vpc.demo.id | ||
| } | ||
| ``` | ||
| 4. **Storing in Database**: | ||
| - After processing, the Harvester stores the identified mappings in a database. | ||
| - The database contains a table like `tf_resource_attribute_mappings`, where the mappings between resource attributes are recorded. | ||
| - This structured storage allows for efficient querying and analysis of the resource attribute mappings. | ||
| ## Usage | ||
| ### Command Syntax | ||
| To use the Terraform Mappings Harvester, run the following command: | ||
| ```sh | ||
| terrarium harvest mappings --dir <path-to-terraform-directory> | ||
| ``` | ||
|
|
||
| Replace `<path-to-terraform-directory>` with the actual path to your Terraform project. | ||
|
|
||
| #### Additional Flags | ||
|
|
||
| - **Module List File** (`--module-list-file`): Specify a file listing the modules to be processed. This is useful for analyzing multiple modules. | ||
|
|
||
| - **Working Directory** (`--workdir`): Define a directory for storing module sources. This improves performance by reusing data between commands, especially beneficial when processing multiple modules. | ||
|
|
||
| ### Direct Scraping | ||
|
|
||
| To scrape mappings directly from a Terraform directory: | ||
|
|
||
| 1. **Run the Harvester**: | ||
|
|
||
| ```sh | ||
| terrarium harvest mappings --dir <path-to-terraform-directory> | ||
| ``` | ||
|
|
||
| This command will parse the Terraform code in the specified directory and identify resource attribute mappings. | ||
|
|
||
| ### Using a Module List File | ||
|
|
||
| To scrape mappings using a module list file: | ||
|
|
||
| 1. **Prepare a Module List File**: Create a file listing the modules you want to process. Refer to the [module list file documentation](https://github.com/cldcvr/terrarium/blob/main/src/pkg/metadata/modulelist/readme.md) for the format and details. | ||
|
|
||
| 2. **Run the Harvester**: | ||
|
|
||
| ```sh | ||
| terrarium harvest mappings --module-list-file <path-to-module-list-file> | ||
| ``` | ||
|
|
||
| This command will process only the modules specified in the module list file. | ||
|
|
||
| ## Monitoring Execution | ||
|
|
||
| Monitor the harvester's execution through the console output. It will provide progress messages and any errors encountered during the scraping process. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| # Terraform Modules Harvester | ||
|
|
||
| The Terraform Modules Harvester is a tool designed to scrape Terraform modules and their attributes from a specified directory or a list of modules. | ||
|
|
||
| ## Working | ||
|
|
||
| ```mermaid | ||
| flowchart TD | ||
| TC["Terraform HCL code"] --"terraform init"--> T[".terraform/modules"] | ||
| T --> MI["Module names, sources, versions\n.terraform/modules/modules.json"] | ||
| T --> MA["Module attributes\n.terraform/modules/module_name/*.tf"] | ||
| MI --> MH["Module Harvester"] | ||
| MA --> MH | ||
| MH --> DB["Database\ntf_modules\ntf_module_attributes"] | ||
| ``` | ||
|
|
||
| The Terraform Modules Harvester operates through a series of steps to extract information about Terraform modules and their attributes. The process is visualized in the above Mermaid flowchart and can be described as follows: | ||
|
|
||
| 1. **Terraform HCL Code Processing**: | ||
| - The process begins with the Terraform HCL (HashiCorp Configuration Language) code. | ||
| - The user runs `terraform init` on their Terraform code. This command initializes a Terraform working directory by downloading and installing the necessary providers and modules. | ||
|
|
||
| 2. **Extraction of Module Information**: | ||
| - The initialization process creates a `.terraform/modules` directory. | ||
| - Within this directory, two key sources of information are generated: | ||
| - **Module Information File (`modules.json`)**: This file, located at `.terraform/modules/modules.json`, contains details about module names, sources, and versions. It acts as a central repository of metadata about the modules used in the Terraform code. | ||
| - **Module Attributes**: For each module, there are HCL files (`.tf` files) within the module's directory (e.g., `.terraform/modules/module_name/*.tf`). These files hold the specific attributes and configurations of each module. | ||
|
|
||
| 3. **Module Harvester Processing**: | ||
| - The Module Harvester tool, processes the information extracted from the `.terraform/modules` directory. | ||
| - It reads both the module information from `modules.json` and the module attributes from the individual `.tf` files. | ||
| - This step translates the raw data from the Terraform configuration and module files into structured information. | ||
|
|
||
| 4. **Storing in Database**: | ||
| - Once the Module Harvester has processed the information, it stores the results in a database. | ||
| - The database consists of tables like `tf_modules` and `tf_module_attributes`, where the harvested data about modules and their attributes is organized and stored for further use or analysis. | ||
|
|
||
| ## Usage | ||
|
|
||
| ### Modes of Operation | ||
|
|
||
| The scraper can be operated in two distinct modes: | ||
|
|
||
| 1. **Direct Scraping from Terraform Directory**: This mode directly scrapes modules from a specified Terraform directory. | ||
|
|
||
| 2. **Using a Module List File**: This mode processes modules listed in a provided module list file. | ||
|
|
||
| ### Prerequisites | ||
|
|
||
| - **Terraform**: Ensure Terraform is installed and accessible. | ||
|
|
||
| ### Direct Scraping | ||
|
|
||
| To scrape modules directly from a Terraform directory: | ||
|
|
||
| 1. **Initialize Terraform**: Ensure you have run `terraform init` in your Terraform project directory. | ||
|
|
||
| 2. **Run the Scraper**: | ||
|
|
||
| ```sh | ||
| terrarium harvest modules --dir <path-to-terraform-directory> | ||
| ``` | ||
|
|
||
| Replace `<path-to-terraform-directory>` with the actual path to your Terraform project. | ||
|
|
||
| #### Additional Flags | ||
|
|
||
| - **Enable Local Modules** (`--enable-local-modules`): Include locally referenced modules in the scraping process with project directory path as the namespace. This is a boolean flag. | ||
|
|
||
| ### Using a Module List File | ||
|
|
||
| To scrape modules using a module list file: | ||
|
|
||
| 1. **Prepare a Module List File**: Create a file listing the modules you want to process. Refer to the [module list file documentation](https://github.com/cldcvr/terrarium/blob/main/src/pkg/metadata/modulelist/readme.md) for the format and details. | ||
|
|
||
| 2. **Run the Scraper**: | ||
|
|
||
| ```sh | ||
| terrarium harvest modules --module-list-file <path-to-module-list-file> | ||
| ``` | ||
|
|
||
| Replace `<path-to-module-list-file>` with the path to your module list file. | ||
|
|
||
| #### Additional Flags | ||
|
|
||
| - **Working Directory** (`--workdir`): Specify a directory for storing module sources. This improves performance by reusing data between running commands for the multiple modules in the list file. | ||
|
|
||
| ### Monitoring Execution | ||
|
|
||
| Monitor the scraper's execution through the console output. It will provide progress messages and any errors encountered during the scraping process. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
5dc63c5There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
5dc63c5There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
5dc63c5There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.