diff --git a/devops/providers/azure-devops/templates/infrastructure/README.md b/devops/providers/azure-devops/templates/infrastructure/README.md index 6ab03c95..f70675ce 100644 --- a/devops/providers/azure-devops/templates/infrastructure/README.md +++ b/devops/providers/azure-devops/templates/infrastructure/README.md @@ -207,7 +207,7 @@ This group has to be named `$STAGE Environment Variables`. It will need the foll | Variable Name | Description | Sample Value | | ------------- | ------------- | ------------- | -| `ARM_SUBSCRIPTION_ID` | The Azure subscription of the service principal used for the deployment | 49e3a7a8-c63e-4124-948c-ee51b74d5801 | +| `ARM_SUBSCRIPTION_ID` | The Azure subscription of the service principal used for the deployment | 98z7y6x5-w43v-2198-765u-ts43r21q9876 | | `REMOTE_STATE_ACCOUNT` | The Azure storage account for remote terraform state | cobaltfstate | | `SERVICE_CONNECTION_NAME` | The azure devops service connection name to use for the Terraform deployments | Cobalt Deployment Administrator | | `TF_CLI_ARGS` | specify additional arguments to the command-line. This allows easier automation in CI environments as well as modifying default behavior of Terraform. If nothing is passed in to this variable, Terraform's default behavior will take place | -refresh=false | diff --git a/docs/GETTING_STARTED_ADD_PAT_OWNER.md b/docs/GETTING_STARTED_ADD_PAT_OWNER.md index 761b68e9..25ead969 100644 --- a/docs/GETTING_STARTED_ADD_PAT_OWNER.md +++ b/docs/GETTING_STARTED_ADD_PAT_OWNER.md @@ -1,353 +1,205 @@ # Getting Started - Advocated Pattern Owner -## Cobalt Enterprise Integration - Overview - -Completion of the steps from this document results in an Azure Devops Repo initialized with carefully selected Cobalt Advocated Pattern Templates (Infrastructure as code) along with a CI/CD pipeline ready for multi-stage deployments. - -This document provides Cobalt users instructions for initializing and integrating Cobalt into their existing AzureDevops organization using an Azure subscription. These steps assume some basic familiarity with the Azure DevOps portal and a desire to automate the creation of infrastructure. For more information on Cobalt, visit the following link: [READ ME](../README.md) - -### Prerequisites - - * An Azure Subscription - * Azure Devops Organization - * Permissions to your Organization's Azure Devops account - * *Global administrator role* permissions in your Organization's Azure Active Directory tenant to setup service principals - * If this is not allowed by your organization, step two and the Service Connection creation in step three will need to be completed by someone within your organization with this permission. - -> If using the CLI commands please set the following variables: - -| Variable | Sample Value | Description | -|----------|--------------|-------------| -| `TEMPLATE_DEVOPS_PROJECT_NAME` | `My Application` | The name of the project representing your Cobalt template application that serves as your organization's advocated pattern for a specific template. | -| `TEMPLATE_DEVOPS_INFRA_REPO_NAME` | `az-hello-world` | The name of the repo that will be created in the application Azure DevOps project to host the Cobalt template. | -| `TEMPLATE_DEVOPS_INFRA_YML_PATH` | `devops/providers/azure-devops/templates/azure-pipelines.yml` | The path relative to the `TEMPLATE_DEVOPS_INFRA_REPO_NAME` root that contains the Cobalt template pipeline to be created for testing and provisioning resources. | -| `DEFAULT_ORGANIZATION` | `https://dev.azure.com/MyOrganization/` | The full URL path of the organization in which your `TEMPLATE_DEVOPS_PROJECT_NAME` resides or will be created. | -| `COBALT_SOURCE_URL` | `https://github.com/microsoft/cobalt.git` | The Git clone URL for Cobalt (containing all templates including the one to be targeted by this template repository) from which this Cobalt template repository will be sourced. | - -Update these values for your environment and application based on the guidance in the table above. -```bash -export TEMPLATE_DEVOPS_PROJECT_NAME="" -export TEMPLATE_DEVOPS_INFRA_REPO_NAME="" -export TEMPLATE_DEVOPS_INFRA_YML_PATH="" -export DEFAULT_ORGANIZATION="" -export COBALT_SOURCE_URL="" -export SUBSCRIPTION_ID="" -export SUBSCRIPTION_NAME="" -export TENANT_ID="" -export SERVICE_PRIN_ID="" -export SERVICE_CONN_NAME="" -export AGENT_POOL_NAME="" -``` - -The following values are used like constants and should not need to change (unless the build pipeline definition is modified). - -```bash -export COBALT_VAR_GROUP_INFRA="Infrastructure Pipeline Variables" -export COBALT_VAR_GROUP_ENV_SUFFIX="Environment Variables" -export COBALT_PIPELINE_NAME="Cobalt CICD Pipeline" -``` +*Prefer using cli? Follow the [cli-based walkthrough](./GETTING_STARTED_APP_PAT_OWNER_CLI.md).* -### STEPS +## Overview -1. **Initialize Azure Repo Subscription with Cobalt** +The goal of this document is to leverage Azure DevOps to operationalize Cobalt Infrastructure Templates. Completion of these steps results in a configured source code management flow and unified pipeline within Azure DevOps ready for multi-stage environments in Azure. - This step helps setup your Azure Devops repo with Cobalt Advocated Pattern Templates that matter to you. These are common instructions that are needed for any audience interested in using Cobalt for infrastructure automation. +This section provides Cobalt users instructions for initializing and integrating Cobalt into their existing Azure DevOps organization using an Azure subscription. These steps assume some basic familiarity with the Azure DevOps portal, the Azure Cloud portal and a desire to automate the creation of infrastructure. For more information on Cobalt, visit the following link: [READ ME](../README.md) - * Create a new project - * Sign-in to Azure DevOps (https://azure.microsoft.com/en-us/services/devops/) - * Select New project and create a name. (ex. Cobalt-Contoso) +## Prerequisites + + 1. An Azure Subscription + 2. Azure DevOps Organization + 3. Permissions to your Organization's Azure DevOps account + 4. An Azure Active Directory Application with a *Global administrator role* permission in your Organization's Tenant. This role is granted the right to create service principals. + +> NOTE: If this is not allowed by your organization tenant, completion of step two and the Service Connection creation in step three will need to be completed by someone within your organization with this permission. + +### STEPS - ![New Project](https://user-images.githubusercontent.com/10041279/63442791-4f868600-c3f9-11e9-91f3-c959654f5a1c.png) +1. **Initialize Azure DevOps Repo with Cobalt** - * Select Create + The following steps help setup an Azure DevOps repo with Cobalt Infrastructure Templates that you can host and deploy. The end result is a cloned Cobalt repo that also comes with a ready to be integrated pipeline configuration file needed for automating your infrastructure deployments in Azure DevOps. -> The following CLI command(s) can be run as an alternative to using the portal-based instructions: + * Create a new project + 1. Sign-in to Azure DevOps (https://azure.microsoft.com/en-us/services/devops/) + 1. Create new project under Organization. First create new Organization if it doesn't exist. (ex. Cobalt-``) - -```bash -az devops configure --defaults organization="$DEFAULT_ORGANIZATION" -az devops project create --name "$TEMPLATE_DEVOPS_PROJECT_NAME" --source-control git --visibility private -az devops configure -d project="$TEMPLATE_DEVOPS_PROJECT_NAME" -``` + ![alt text](./images/Org.png "Create Organization") + ![alt text](./images/project.png "Create Project") -* Create new repository by fetching source code from the master branch of Cobalt's open-source github project - * Select Repos tab within side-navigation menu - * Select 'Import a repository' from the Repos tab sub-menu and click [Import] - * Enter the Cobalt Clone URL (https://github.com/microsoft/cobalt.git) and select Import + * Create new repository by fetching source code from the master branch of Cobalt's open-source github project. This will allow you to host and grow your own repo for Cobalt Infrastructure Templates. + 1. Select Repos tab within side-navigation menu + 1. Select 'Import a repository' from the Repos tab sub-menu and click [Import] + 1. Enter the Cobalt Clone URL (https://github.com/microsoft/cobalt.git) and select Import - ![Clone Button](https://user-images.githubusercontent.com/10041279/63459072-8ec4cf00-c419-11e9-8ef4-ee7db827e49c.png) + ![Clone Button](https://user-images.githubusercontent.com/10041279/63459072-8ec4cf00-c419-11e9-8ef4-ee7db827e49c.png) - * Rename your repository - * Click your Repo Name (ex. Cobalt-Contoso) at the top of the page. + * Personalize the project by renaming your cloned Cobalt repo. + 1. Click your Repo Name (ex. Cobalt-``) at the top of the page. ![Repo Name dropdown](https://user-images.githubusercontent.com/10041279/63615290-9d8ebb80-c5aa-11e9-9136-64295640205b.png) - * Select Manage Repositories - * Under Git repositories, find "Cobalt-Contoso" and select the ellipses - * Select Rename repository + 1. Select Manage Repositories + 1. Under Git repositories, find "Cobalt-``" and select the ellipses + 1. Select Rename repository ![Rename Repo dropdown](https://user-images.githubusercontent.com/10041279/63615540-3b828600-c5ab-11e9-9174-07f23b3193bf.png) - * Give it a new name and click [Rename] + 1. Give it a new name and click [Rename] | Naming Recommendation | Template Repo Strategy | |-------------|-----------| - | Cobalt-Hello-World-Contoso | If the aim is to introduce oneself or the organization to Cobalt, we recommended a name that reflects the spirit of the Azure Hello World Cobalt template. | - | Cobalt-AZ-ISO-Contoso | If the aim is to have a single repository represent a single Cobalt template, and thereafter, to have one repo per template, we recommend a name that reflects the Cobalt Template being deployed. In this naming example, the name assumes this repo will be dedicated to deploying the Cobalt *az-isolated-service-single-region* template | - | Cobalt-Contoso | If the aim is to use a single repository as ground truth for all potential patterns across your organization, effectively having to manage a combination of Cobalt patterns from a single repo, it's recommended to stick with a name that matches the project name. | - - -> The following CLI command(s) can be run as an alternative to using the portal-based instructions: - -```bash -az repos create --name "$TEMPLATE_DEVOPS_INFRA_REPO_NAME" -az repos import create --git-url $COBALT_SOURCE_URL --repository "$TEMPLATE_DEVOPS_INFRA_REPO_NAME" -``` - -* Initialize new Azure Devops pipeline - * Select Pipelines tab from within side-navigation menu - * Select Create Pipeline and then choose 'Azure Repos Git [YAML]' - - ![Pipeline Menu](https://user-images.githubusercontent.com/10041279/63459652-89b44f80-c41a-11e9-829a-05a6888b7673.png) + | Cobalt-Hello-World-`` | If the aim is to introduce oneself or the organization to Cobalt, we recommended a name that assumes this repo will only be hosting the Azure Hello World Cobalt Infrastructure Template. | + | Cobalt-AZ-ISO-`` | If already familiar with Cobalt and the aim is to have a single repository represent a single Cobalt Infrastructure Template, and thereafter, to have one repo per template, we recommend a name that reflects the Cobalt Infrastructure Template being deployed. In this naming example, the name assumes this repo will be dedicated to deploying the Cobalt *az-isolated-service-single-region* Infrastructure Template | + | Cobalt-`` | If already familiar with Cobalt ad the aim is to use a single repository as ground truth for all potential patterns across your organization, effectively having to manage a combination of Cobalt patterns from a single repo, it's recommended to stick with a name that matches the project name. | - * Find and select the newly created repository from dropdown menu - * Import YAML by selecting 'Existing Azure Pipelines YAML file' - * Enter the path to the devops yaml file that lives within your newly created repo. (i.e. devops/providers/azure-devops/templates/azure-pipelines.yml) + * Initialize a new Azure DevOps pipeline by integrating an existing YAML file. This file already lives within the newly created repo and is needed as Azure DevOps uses it to orchestrate your Cobalt Infrastructure Template deployments across various deployment stages. By integrating this file, a unified deployment pipeline is executed for you. + 1. Select Pipelines tab from within side-navigation menu + + ![alt text](./images/pipline.png "Create Pipline") - ![Select YAML](https://user-images.githubusercontent.com/10041279/63459938-21b23900-c41b-11e9-9b9c-2dfa72e51350.png) + 1. Select New Pipeline and then choose 'Azure Repos Git [YAML]' - > NOTE: Automatic drop-down does not always populate with yaml file options. It may be necessary to simply copy and paste the above path. - * Review devops pipeline YAML and only keep templates relevant to your enterprise patterns. - * Remove jobName configurations not relevant to your enterprise patterns. If new to Cobalt, we recommend keeping the path to the az_hello_world template as a starter template. This step can also be completed later as a code commit to your repo. Below is an example of a jobName that you may want to remove by simple deleting it. - ```yaml - configurationMatrix: - - jobName: az_service_single_region - terraformTemplatePath: 'infra/templates/az-service-single-region' - terraformWorkspacePrefix: 'sr' - environmentsToTeardownAfterRelease: - - 'devint' - ``` - * Save and run + ![Pipeline Menu](https://user-images.githubusercontent.com/10041279/63459652-89b44f80-c41a-11e9-829a-05a6888b7673.png) - ![Fail Screenshot](https://user-images.githubusercontent.com/10041279/63546484-8ccd3f80-c4ef-11e9-8d9f-2f06dc725fc7.png) + 1. Find and select the newly created repository from dropdown menu + 1. Import YAML by selecting 'Existing Azure Pipelines YAML file' + * Enter the path to the devops yaml file that lives within your newly created repo. (i.e. devops/providers/azure-devops/templates/azure-pipelines.yml) - > NOTE: Azure Devops forces a run so expect this to fail. Future steps will resolve this problem. + ![Select YAML](https://user-images.githubusercontent.com/10041279/63459938-21b23900-c41b-11e9-9b9c-2dfa72e51350.png) + > NOTE: Automatic drop-down does not always populate with yaml file options. It may be necessary to simply copy and paste the above path. + 1. Review the devops pipeline YAML file and only keep references to Cobalt Infrastructure Templates that your organization would like to deploy. This step helps configure the Azure DevOps project to rely on this file in order to build and deploy the referenced Cobalt Infrastructure Templates across various deployment stages. + * Remove jobName configurations that reference Cobalt Infrastructure Templates not intended for deployment. If new to Cobalt, we recommend keeping the jobName that references our az_hello_world infrastructure template as a starter template. This step can also be completed later as a code commit to your repo. Below is an example of a jobName that you may want to remove by simply deleting the section from the file. + ```yaml + configurationMatrix: + - jobName: az_service_single_region + terraformTemplatePath: 'infra/templates/az-service-single-region' + terraformWorkspacePrefix: 'sr' + environmentsToTeardownAfterRelease: + - 'devint' + ``` + 1. Save and run -> The following CLI command(s) can be run as an alternative to using the portal-based instructions: + ![Fail Screenshot](https://user-images.githubusercontent.com/10041279/63546484-8ccd3f80-c4ef-11e9-8d9f-2f06dc725fc7.png) - ``` - az pipelines create --name "$COBALT_PIPELINE_NAME" --repository "$TEMPLATE_DEVOPS_INFRA_REPO_NAME" --branch master --repository-type tfsgit --yml-path $TEMPLATE_DEVOPS_INFRA_YML_PATH --skip-run true - ``` + > NOTE: Azure Devops forces a run so expect this to fail. Future steps will resolve this problem. 2. **Provision Azure resources needed for Azure Devops pipeline** - This step sets up all the values and resources that will serve as inputs to your test automation pipeline in Azure Devops. Without this setup step, you cannot deploy Cobalt templates to Azure Devops. + Follow this step to setup Azure resources (i.e. Azure Blob Storage, etc.) in the appropriate Azure Subscription. Later, these resources will need to be referenced in your Azure DevOps test automation pipeline in order to make the pipeline fully functional. Without this step, you cannot deploy Cobalt Infrastructure Templates to Azure DevOps. * Create a registered Azure AD (AAD) app for Cobalt deployments - * Sign-in to your organization's Azure account. (https://portal.azure.com) - * Filter for Azure Active Directory and navigate to it's menu - * Select App registrations from the menu blade - * Click [Add/+] New registration then enter a name for the application (ex. cobalt-hw-admin-sp-`` or cobalt-az-iso-admin-sp-``) - * Choose single tenant as a supported account type - * Click Register + 1. Sign-in to your organization's Azure account. (https://portal.azure.com) + 2. Filter for Azure Active Directory and navigate to it's menu + 3. Select App registrations from the menu blade + + ![alt text](./images/AppReg.png "App Registrations") + + 4. Click [Add/+] New registration then enter a name for the application (ex. cobalt-hw-admin-sp-`` or cobalt-az-iso-admin-sp-``) + 5. Choose single tenant as a supported account type + 6. Click Register * Setup permissions for the new AAD app to also use legacy API permissions - * From the App registrations service blade, select the API permissions - * Click [+ Add a permission] then use the *APIs my Organization uses* tab to search for Windows Azure Active Directory - * Configure Azure Activity Directory Application permissions to ReadWrite.OwnedBy + 1. From the App registrations service blade, select the API permissions + + ![alt text](./images/api-permissions.png "API Permissions") + + 3. Configure Azure Activity Directory Application permissions to ReadWrite.OwnedBy. First select Microsoft Graph and then select Application Permissions ![Request Permissions menu](https://user-images.githubusercontent.com/10041279/63549279-b6896500-c4f5-11e9-9c92-40ac2a4295c9.png) - * Click [Add permissions] to save this configuration - * Click [Grant admin consent for *Your Directory*] to grant consent on behalf of users in this directory for this permission + 4. Click [Add permissions] to save this configuration + 5. Click [Grant admin consent for *Your Directory*] to grant consent on behalf of users in this directory for this permission * Configure the new AAD app as a Cobalt admin service-principal/service-endpoint - * From the App registrations service blade, click the [Certificates & secrets] tab - * Click [+New client secret] from within the Client secrets menu then enter a description (ex. rbac) + 1. From the App registrations service blade, click the [Certificates & secrets] tab + 2. Click [+New client secret] from within the Client secrets menu then enter a description (ex. rbac) ![Client Secret menu](https://user-images.githubusercontent.com/10041279/63461963-69d35a80-c41f-11e9-8d4a-c72235177fb3.png) - * Click Add + 3. Click Add > IMPORTANT: Generate a secret that does not have a trailing slash. Secrets that lead with a slash (ex."/","\") may cause parsing errors. - > NOTE: Take note of the generated client secret (only displayed once). This will be used for your Azure Devops Service Connection in step 3. - * From the App registrations service blade, select Overview. - > NOTE: Take note of the Application (client) ID. This will also be used for your Azure Devops Service Connection in step 3. + > NOTE: Take note of the generated client secret (only displayed once). This will be used for your Azure DevOps Service Connection in step 3. + 4. From the App registrations service blade, select Overview. + > NOTE: Take note of the Application (client) ID. This will also be used for your Azure DevOps Service Connection in step 3. * Grant newly created Service Principal an Owner role to your preferred enterprise subscription. - This elevates the Service Principal with more permissions so that Terraform can rely on this Service Principal as an Azure user for Cobalt template deployments. + This elevates the Service Principal with more permissions so that Terraform can rely on this Service Principal as an Azure user for Cobalt Infrastructure Template deployments. - * Filter for subscriptions and navigate to the subscriptions list - * Either choose a subscription or create a new one (ex. Cobalt-Contoso-Deployments) - * Select your chosen subscription then select the Access control (IAM) tab from the menu blade. - * Click [+/Add] and select Add role assignment - * From the sub-menu, select 'Owner' as a role from the drop down and search for the newly created Service Principal (i.e. cobalt-hw-admin-sp-`` or cobalt-az-iso-admin-sp-``) + 1. Filter for subscriptions and navigate to the subscriptions list + 2. Either choose a subscription or create a new one (ex. Cobalt-``-Deployments) + 3. Select your chosen subscription then select the Access control (IAM) tab from the menu blade. + 4. Click [+/Add] and select Add role assignment + 5. From the sub-menu, select 'Owner' as a role from the drop down and search for the newly created Service Principal (i.e. cobalt-hw-admin-sp-`` or cobalt-az-iso-admin-sp-``) - ![Role Assignment menu](https://user-images.githubusercontent.com/31149154/63708249-7ed23400-c7f9-11e9-8dbb-c15dcdaf3a37.png) + ![Role Assignment menu](https://user-images.githubusercontent.com/31149154/63708249-7ed23400-c7f9-11e9-8dbb-c15dcdaf3a37.png) - * Click Save + 6. Click Save * Create Resource Group and Storage Account for backend state - * Filter for Storage accounts and navigate to the storage account list - * Click [+/Add] and enter values for the following fields: + 1. Filter for Storage accounts and navigate to the storage account list + 2. Click [+/Add] and enter values for the following fields: * Subscription: Your preferred enterprise subscription for Cobalt template deployments * Resource group: Create new (ex. cobalt-devint-hw-admin-rg or cobalt-devint-az-iso-admin-rg ) * Storage account name: (ex. cobalttfstates) - * Click [Review+Create] then [Create] - * Once deployment for storage account is completed, go to the resource and visit the Blobs sub-menu - * Click [+Container] then create a container name (ex. az-hw-remote-state-container or az-iso-remote-state-container) with private access - -> The following CLI command(s) can be run as an alternative to using the portal-based instructions. Note this creates the Service Connection within Azure DevOps, but you will still need to setup the Service Principal: + 3. Click [Review+Create] then [Create] + 4. Once deployment for storage account is completed, go to the resource and visit the Blobs sub-menu + 5. Click [+Container] then create a container name (ex. az-hw-remote-state-container or az-iso-remote-state-container) with private access -```bash -az devops service-endpoint azurerm create --azure-rm-subscription-id $SUBSCRIPTION_ID --azure-rm-subscription-name "$SUBSCRIPTION_NAME" --azure-rm-tenant-id $TENANT_ID --azure-rm-service-principal-id $SERVICE_PRIN_ID --name "$SERVICE_CONN_NAME" -``` +3. **Configure Azure DevOps pipeline by referencing Azure resources created in your Azure Cloud Subscription** -3. **Configure Azure Devops pipeline using Azure resource values** - - This step is about making sure Azure Devops references all the values and resources you took the time to create in the Azure portal. + The goal of this step is to have the Azure Devops pipeline referencing the appropriate Azure resources (i.e. Azure Blob Storage Account, etc.) living within your Azure Cloud Subscription. This will ensure that infrastructure deployments are impacting the desired Azure Cloud Subscription and that the deployment process is permissioned to perform various operations needed for proper build and integration across various deployment stages. * Add the Azure Subscription being used for Cobalt as a *Service Connection* - * Return to your Azure DevOps subscription - * Find and select the Project Settings tab at the bottom of the screen - * Under the Pipelines menu select Service Connections - * From the Service Connections menu, select [+New Service Connection] - * Choose Azure Resource Manager from the dropdown then a name for your service (ex. Cobalt Deployment Administrator-``). The name should make sense to users and will be directly referenced in pipeline variable groups later. - * Use the full version of the service connection dialog in order to enter your service principal credentials (AAD Key, AAD App ID, Tenant, etc.) + 1. Return to your Azure DevOps subscription + 2. Find and select the Project Settings tab at the bottom of the screen + 3. Under the Pipelines menu select Service Connections + 4. From the Service Connections menu, select [+New Service Connection] + 5. Choose Azure Resource Manager from the dropdown then a name for your service (ex. Cobalt Deployment Administrator-``). The name should make sense to users and will be directly referenced in pipeline variable groups later. + 6. Use the full version of the service connection dialog in order to enter your service principal credentials (AAD Key, AAD App ID, Tenant, etc.) ![Service Connection menu](https://user-images.githubusercontent.com/10041279/63485304-63b59c00-c468-11e9-8e47-721a2e43ecb9.png) - * Verify and Save the connection + 7. Verify and Save the connection > NOTE: Take note of the custom name given to this service connection. This will be referenced in later steps needed to configure env variable groups. * Enable multi-stage pipelines - * Find your signed-in avatar/image and select preview features from the drop down menu + 1. Find your signed-in avatar/image and select preview features from the drop down menu - ![Preview Features menu](https://user-images.githubusercontent.com/10041279/63486065-8eedba80-c46b-11e9-90f0-7f931f909ffa.png) + ![Preview Features menu](./images/preview.png) - * Toggle Multi-stage pipelines + 2. Toggle Multi-stage pipelines * Configure *Infrastructure Pipeline Variables* as the first of two variable groups - * Select Pipelines tab from within side-navigation menu then select Library tab - * Click [+Variable group] and name it "Infrastructure Pipeline Variables" - * Add the following variables: - - | Name | Value | Var Description | - |-------------|-----------|-----------| - | `AGENT_POOL` | Hosted Ubuntu 1604 | The type of build agent used for your deployment. | - | `ARM_PROVIDER_STRICT` | false | Terraform ARM provider modification | - | `BUILD_ARTIFACT_NAME` | drop | Name to identity the folder containing artifacts output by a build. | - | `GO_VERSION`| 1.12.5 | The version of Go terraform deployments are bound to. | - | `PIPELINE_ROOT_DIR` | devops/providers/azure-devops/templates/ | A path for finding Cobalt templates. | - | `REMOTE_STATE_CONTAINER` | ``| The remote blob storage container name for managing the state of a Cobalt Template's deployed infrastructure. Also is used as a naming convention for partitioning state into multiple workspaces. This name was created in an earlier step from within the azure portal. | - | `SCRIPTS_DIR` | infrastructure/scripts | Path to scripts used at runtime for composing build and release jobs at various pipeline stages. | - | `TEST_HARNESS_DIR` | test-harness/ | A path to the cobalt test harness for running integration and unit tests written in Docker and Golang. | - | `TF_ROOT_DIR`| infra | The primary path for all Cobalt templates and the modules they are composed of. | - | `TF_VERSION`| 0.12.4 | The version of terraform deployments are bound to. | - | `TF_WARN_OUTPUT_ERRORS`| 1 | The severity level of errors to report. | - - > Important: Every targeted environment specified within the build pipeline expects a - > variable group specified with the naming convention ` Environment Variables` - -> The following CLI command(s) can be run as an alternative to using the portal-based instructions: - -```bash -# IMPORTANT: Replace these values as necessary to fit your environment. -az pipelines variable-group create --authorize true --name "$COBALT_VAR_GROUP_INFRA" --variables \ - AGENT_POOL="$AGENT_POOL_NAME" \ - ARM_PROVIDER_STRICT=true \ - BUILD_ARTIFACT_NAME='drop' \ - BUILD_ARTIFACT_PATH_ALIAS='artifact' \ - GO_VERSION='1.12.5' \ - PIPELINE_ROOT_DIR='devops/providers/azure-devops/templates/infrastructure' \ - REMOTE_STATE_CONTAINER='BACKENDSTATECONTAINERNAME' \ - SCRIPTS_DIR='scripts' \ - TEST_HARNESS_DIR='test-harness/' \ - TF_DEPLOYMENT_TEMPLATE_ROOT='infra/templates/$TEMPLATE_DEVOPS_INFRA_REPO_NAME' \ - TF_DEPLOYMENT_WORKSPACE_PREFIX='PROJECTDEPLOYMENTWORKSPACEPREFIX' \ - TF_ROOT_DIR='infra' \ - TF_VERSION='0.12.4' \ - TF_WARN_OUTPUT_ERRORS=1 -``` - -* Configure *DevInt Environment Variables* as the final variable group - * Environment-specific variables have no default values and must be assigned - * Return to the Library tab - * Click [+Variable group] and name it *DevInt Environment Variables* - * Add the following variables: - - | Name | Value | Var Description | - |-------------|-----------|-----------| - | `ARM_SUBSCRIPTION_ID` | `` | The Azure subscription ID for which all resources will be deployed. Refer to the Azure subscription chosen in Azure portal for Cobalt deployments. | - | `REMOTE_STATE_ACCOUNT` | `` | The storage container account name created in a previous step that is used to manage the state of this deployment pipeline. The storage Account is shared among all non-prod deployment stages. | - | `SERVICE_CONNECTION_NAME` | ex. Cobalt Deployment Administrator-`` | The custom name of the service connection configured in a previous Azure Devops step that establishes a connection between the Service Principal and the Azure subscription that it's permissioned for. | - | `TF_CLI_ARGS` | "-refresh=false" | specify additional arguments to the command-line. This allows easier automation in CI environments as well as modifying default behavior of Terraform | - -> The following CLI command(s) can be run as an alternative to using the portal-based instructions: - -```bash -# IMPORTANT: Replace these values as necessary to fit your environment. -DEVINT_VAR_GROUP="DevInt $COBALT_VAR_GROUP_ENV_SUFFIX" -az pipelines variable-group create --authorize true --name "$DEVINT_VAR_GROUP" --variables \ - ARM_SUBSCRIPTION_ID='TARGETSUBSCRIPTIONID' \ - REMOTE_STATE_ACCOUNT='BACKENDSTATESTORAGEACCOUNTNAME' \ - SERVICE_CONNECTION_NAME='SERVICECONNECTIONNAME' \ - TF_CLI_ARGS='-refresh=false' -``` - -* Additional Setup Instructions per Template - - Select Cobalt templates require additional pipeline setup. Please complete extended steps if chosen template resides in the below list. - - * az-isolated-service-single-region - 1. Create ASE w/ VNET - 2. Add additional env vars to *Infrastructure Pipeline Variables* group - - | Name | Value | Var Description | - |-------|-------|-----------------| - | `TF_DEPLOYMENT_TEMPLATE_ROOT` | infra/templates/az-isolated-service-single-region | Pipeline reference for relative location of this template | - -* Link Variable Groups for DevInt and Infrastructure to the Build Pipeline - * Select Pipelines tab from within side-navigation menu - * Select existing pipeline and then click [Edit] - * Next to the [Variables] button at the top of the page, click the ellipses and select Triggers - - ![Triggers](https://user-images.githubusercontent.com/41071421/63284806-022fda80-c27a-11e9-8e23-494314c63651.png) - - * Navigate to the [Variables] tab and begin linking each variable group - * Link each variable group, one by one - - ![Link Variable Groups](https://user-images.githubusercontent.com/10041279/63489261-3b816980-c477-11e9-87bf-1d254226e8fd.png) - - * Save the build pipeline - -> The following CLI command(s) can be run as an alternative to using the portal-based instructions: - -```bash -az pipelines show --name "$COBALT_PIPELINE_NAME" -o json > builddef.json -PIPELINE_ID=$(az pipelines show --name "$COBALT_PIPELINE_NAME" --query id) -``` - -Execute the list command to find the Variable Group IDs created earlier. Make note of the IDs as they will need to be added to the build pipeline definition. - -```bash -az pipelines variable-group list -``` - -For the workaround, you'll be manually editing the builddef.json file to add the variable group references. At the end of the file, you should see the line `"variableGroups" : null`. Replace the value with the following, replacing the variable group ID placeholders (`0`) with those from the above command for the Infrastructure Pipeline Variables group and DevInt Environment Variables group: -```bash -"variableGroups": [ - { "id": 0 }, - { "id": 0 } -], -``` - -```bash -az devops invoke --http-method PUT --area build --resource definitions --route-parameters project="$TEMPLATE_DEVOPS_PROJECT_NAME" definitionId=$PIPELINE_ID --in-file builddef.json - -``` + 1. Select Pipelines tab from within side-navigation menu then select Library tab + 2. Click [+Variable group] and name it "Infrastructure Pipeline Variables" + 3. Create a list of required variables. These variables can be found by visiting the *[Gloal Variable Group](./../devops/providers/azure-devops/templates/infrastructure/README.md#Global-Variable-Group)* section of our Azure DevOps infrastructure readme. + + * Configure *DevInt Environment Variables* as the final variable group + 1. Environment-specific variables have no default values and must be assigned + 2. Return to the Library tab + 3. Click [+Variable group] and name it *DevInt Environment Variables* + 4. Create a list of required variables. These variables can be found by visiting the *[Stage Specific Pipeline Variables](./../devops/providers/azure-devops/templates/infrastructure/README.md#Stage-Specific-Pipeline-Variables)* section of our Azure DevOps infrastructure readme. + + * Link Variable Groups for DevInt and Infrastructure to the Build Pipeline + 1. Select Pipelines tab from within side-navigation menu + 2. Select existing pipeline and then click [Edit] + 3. Next to the [Variables] button at the top of the page, click the ellipses and select Triggers + + ![Triggers](https://user-images.githubusercontent.com/41071421/63284806-022fda80-c27a-11e9-8e23-494314c63651.png) + + 4. Navigate to the [Variables] tab and begin linking each variable group + 5. Link each variable group, one by one + + ![Link Variable Groups](https://user-images.githubusercontent.com/10041279/63489261-3b816980-c477-11e9-87bf-1d254226e8fd.png) + + 6. Save the build pipeline 4. **Clone newly created Azure DevOps Repo from your organization** @@ -363,21 +215,20 @@ az devops invoke --http-method PUT --area build --resource definitions --route-p 5. **Keep the templates relevant to your enterprise patterns** - The goal of this step is to continue efforts removing infrastructure as code Cobalt templates that users have no interest in deploying. + The goal of this step is to continue efforts removing Cobalt Infrastructure Template directories that users have no interest in deploying. * Open the project from your favorite IDE and navigate to infrastructure templates `./infra/templates` directory. - * Manually delete template directories not needed for your enterprise. - > NOTE: Do not delete 'backend-state-setup' template! We also recommended keeping the 'az-hello-world' template as a starter template. + * Manually delete template directories not needed for your enterprise. (Do not delete 'backend-state-setup' template! We also recommended keeping the 'az-hello-world' template as a starter template.) + + ![image](https://user-images.githubusercontent.com/10041279/64913136-1d1e2f00-d700-11e9-95cd-9e95c257bcbd.png) + * Commit the newly pruned project to your newly forked repo. ```bash $ git commit -m "Removed unrelated templates." && git push ``` - > NOTE: Integration tests running in the release stage of the pipeline may have resource group level naming conflicts if other tests of the same templates are also running or have been persisted in the Azure portal. -```bash -az pipelines run --name "$COBALT_PIPELINE_NAME" -``` + > NOTE: The CI/CD pipeline needs to detect a code change to run the template-specific build and release jobs (in their respective stages). To force the template build and release to run, you may add a `FORCE_RUN` environment variable with a value of `true` to your *devint Environment Variables* variable group. You may also add a comment or extra line to a TF or Go file within the template in order for the pipeline script to detect a change without adding any additional override flags. -## Conclusion +## Additional Recommendations - Recommended next step is to either reference containerized applications by their image name from within a Cobalt template in order to run a deployment or to employ this repo as ground truth for acceptable patterns and versioning across an organization. +Recommended next step is to target containerized applications via their image names from within a Cobalt Infrastructure Template. diff --git a/docs/GETTING_STARTED_ADD_PAT_OWNER_CLI.md b/docs/GETTING_STARTED_ADD_PAT_OWNER_CLI.md new file mode 100644 index 00000000..71211fac --- /dev/null +++ b/docs/GETTING_STARTED_ADD_PAT_OWNER_CLI.md @@ -0,0 +1,268 @@ +# Getting Started - Advocated Pattern Owner - Azure CLI + +*Prefer using portals? Follow the [portal-based walkthrough](./GETTING_STARTED_ADD_PAT_OWNER.md).* + +## Overview + +The goal of this document is to leverage Azure DevOps to operationalize Cobalt Infrastructure Templates. Completion of these steps results in a configured source code management flow and unified pipeline within Azure DevOps ready for multi-stage environments in Azure. + +This section provides Cobalt users instructions for initializing and integrating Cobalt into their existing Azure DevOps organization using an Azure subscription. These steps assume some basic familiarity with the Azure DevOps portal, the Azure Cloud portal and a desire to automate the creation of infrastructure. For more information on Cobalt, visit the following link: [READ ME](../README.md) + +## Prerequisites + + * An Azure Subscription + * Azure DevOps Organization + * Permissions to your Organization's Azure DevOps account + * An Azure Active Directory Application with a *Global administrator role* permission in your Organization's Tenant. This role is granted the right to create service principals. + +> NOTE: If this is not allowed by your organization tenant, these steps will need to be completed by someone within your organization with elevated permissions. + + * Azure CLI + * [Get started with Azure CLI](https://docs.microsoft.com/en-us/cli/azure/get-started-with-azure-cli?view=azure-cli-latest) + * Azure DevOps CLI extension + * [Get started with Azure DevOps CLI](https://docs.microsoft.com/en-us/azure/devops/cli/get-started?view=azure-devops) + +## Steps + +### 1. Setup Environment Variables + +The environment variables set below will be referenced by various cli commands highlighted throughout this install script: + +| Variable | Sample Value | Description | +|----------|--------------|-------------| +| `TEMPLATE_DEVOPS_PROJECT_NAME` | `My Project` | The name of the DevOps project hosting a single or multi-repo Cobalt project.| +| `TEMPLATE_DEVOPS_INFRA_REPO_NAME` | `az-hello-world` | The name of the repo that will be created in the Azure Devops project needed to host the Cobalt Template. | +| `TEMPLATE_DEVOPS_INFRA_YML_PATH` | `devops/providers/azure-devops/templates/azure-pipelines.yml` | The path relative to the `TEMPLATE_DEVOPS_INFRA_REPO_NAME` root that contains the Cobalt Infrastructure Template pipeline to be created for testing and provisioning resources. | +| `DEFAULT_ORGANIZATION` | `https://dev.azure.com/MyOrganization/` | The full URL path of the organization in which your `TEMPLATE_DEVOPS_PROJECT_NAME` resides or will be created. | +| `COBALT_SOURCE_URL` | `https://github.com/microsoft/cobalt.git` | The Git clone URL for Cobalt (containing all templates including those targeted by this AZDO project) from which all Cobalt Infrastructure Template repositories will be sourced. | +| `AGENT_POOL_NAME` | `Hosted Ubuntu 1604` | The type of image to use for CI.| +| `COBALT_PIPELINE_NAME` | `My Pipeline` | The name of the CI/CD pipeline used for Cobalt Infrastructure Template deployments.| + +> NOTE: Next steps provide more guidance for the proper naming of the above environment variables. + +#### a. Determine environment variable values by choosing an Azure Devops (AZDO) Scenario that will host your Cobalt Infrastructure Template + +The below table highlights three primary scenarios to choose from when structuring your AZDO project for Cobalt Infrastructure Templates. This helps set expectations for the AZDO project's intended usage. Choose one of the following scenario: + +| Env. Variable Names | AZDO Org. Scenario | Description | +|-------------|-----------|-----------| +|`TEMPLATE_DEVOPS_PROJECT_NAME`=`Cobalt-Contoso`
`TEMPLATE_DEVOPS_INFRA_REPO_NAME`=`Cobalt-HW-Contoso`
`TEMPLATE_DEVOPS_INFRA_REPO_NAME`=`Cobalt-AZ-ISO-Contoso`
`TEMPLATE_DEVOPS_INFRA_REPO_NAME`=`...` | One Project for All | Not CLI ready | +|`TEMPLATE_DEVOPS_PROJECT_NAME`=`Cobalt-Contoso`
`TEMPLATE_DEVOPS_INFRA_REPO_NAME`=`Cobalt-Contoso`
`COBALT_PIPELINE_NAME`=`Cobalt-Pipeline`| One Project, One Pipeline for All | If the aim is to use a single repository as ground truth for all potential patterns across your organization, effectively having to manage a combination of Cobalt Infrastructure Templates from a single repo, it's recommended to stick with a project name and repo name that reflects that choice.| +|`TEMPLATE_DEVOPS_PROJECT_NAME`=`Cobalt-Hello-World-Contoso`
`TEMPLATE_DEVOPS_INFRA_REPO_NAME`=`Cobalt-HW-Contoso`
`COBALT_PIPELINE_NAME`=`Cobalt-HW-Pipeline` | One Repo per Project| If the aim is to introduce oneself or the organization to Cobalt Infrastructure Templates, we recommend an AZDO project that only hosts the Azure Hello World Cobalt Infrastructure Template along with a project name and repo name that reflects that choice.| + +Here is an additional diagram that maps to the above tables and provides more guidance on naming your AZDO projects for hosting Cobalt Infrastructure Templates: + +![image](https://user-images.githubusercontent.com/10041279/65463074-b940e800-de1c-11e9-8254-db5d93c3cd4b.png) + +#### b. Update these values for your environment and application based on the guidance in the previous steps + +```bash +export TEMPLATE_DEVOPS_PROJECT_NAME="" +export TEMPLATE_DEVOPS_INFRA_REPO_NAME="" +export TEMPLATE_DEVOPS_INFRA_YML_PATH="" +export DEFAULT_ORGANIZATION="" +export COBALT_SOURCE_URL="" +export SUBSCRIPTION_ID="" +export SUBSCRIPTION_NAME="" +export TENANT_ID="" +export AGENT_POOL_NAME="" +export COBALT_PIPELINE_NAME="" +``` + +The following values are used like constants and should not need to change (unless the build pipeline yaml definition is modified). + +```bash +export COBALT_VAR_GROUP_INFRA="Infrastructure Pipeline Variables" +export COBALT_VAR_GROUP_ENV_SUFFIX="Environment Variables" +export COBALT_PIPELINE_NAME="Cobalt CICD Pipeline" +``` + +> NOTE: Every targeted environment specified within the build pipeline expects a variable group specified with the naming convention "` Environment Variables`" + +### 2. Create and Configure Azure DevOps Project + +The following steps help setup your Azure DevOps repo with Cobalt Infrastructure Templates that matter to you. These are common cli instructions that are needed for any audience interested in using Cobalt for infrastructure automation. + +> NOTE: Before you can run Azure DevOps CLI commands, you need to run the login command (`az login` if using AAD/MSA identity else `az devops login` if using PAT token) to setup credentials. Please see https://aka.ms/azure-devops-cli-auth for more information. + +#### a. Run the following commands to create a new project for an Azure DevOps Organization + +The below will create a project and set your organization as the default organization for all subsequent Azure DevOps CLI commands. + +```bash +az devops configure --defaults organization="$DEFAULT_ORGANIZATION" +# NOTE: The 'create' command returns a json response from ADO +az devops project create --name "$TEMPLATE_DEVOPS_PROJECT_NAME" --source-control git --visibility private +az devops configure -d project="$TEMPLATE_DEVOPS_PROJECT_NAME" +``` + +#### b. Run the following commands to fork Cobalt's master repo into your Azure Devops Project + +```bash +az repos create --name "$TEMPLATE_DEVOPS_INFRA_REPO_NAME" +# NOTE: The 'import create' command returns a json response from ADO +az repos import create --git-url $COBALT_SOURCE_URL --repository "$TEMPLATE_DEVOPS_INFRA_REPO_NAME" +``` + +### 3. Provision Azure resources needed for Azure Devops CI/CD Build Pipeline + + Follow this step to setup Azure resources (i.e. Azure Blob Storage, etc.) in the appropriate Azure Subscription. Later, these resources will need to be referenced in your Azure DevOps test automation pipeline in order to make the pipeline fully functional. Without this step, you cannot deploy Cobalt Infrastructure Templates to Azure DevOps. + +*Steps for provisioning the Azure resources can be found using the second step of the [portal-based walkthrough steps](./GETTING_STARTED_ADD_PAT_OWNER.md#steps).* + +### 4. Setup Azure DevOps CI/CD Build Pipeline for Cobalt using Azure resources + +Resource groups, Service Principal and Storage Accounts created in Azure will now need to be used for setting up your Azure DevOps CI/CD build pipeline. + +#### a. Run the following commands to create a new Azure DevOps CI/CD build pipeline + +We are intentionally skipping the initial run since we know it will fail; we need to link the required variables groups to this pipeline, and that will be done later. + +```bash +# NOTE: The following command returns a json response from ADO. +az pipelines create --name "$COBALT_PIPELINE_NAME" --repository "$TEMPLATE_DEVOPS_INFRA_REPO_NAME" --branch master --repository-type tfsgit --yml-path $TEMPLATE_DEVOPS_INFRA_YML_PATH --skip-run true +``` + +#### b. Add the Azure Principle being used for Cobalt as a *Service Connection* + +> NOTE: The Service Principle was created in an earlier step from within the Azure portal. The custom name chosen to represent the Service Connection will be referenced in later steps needed to configure env variable groups. + +```bash +export SERVICE_PRIN_ID="" +export SERVICE_CONN_NAME="" +``` + +```bash +az devops service-endpoint azurerm create --azure-rm-subscription-id $SUBSCRIPTION_ID --azure-rm-subscription-name "$SUBSCRIPTION_NAME" --azure-rm-tenant-id $TENANT_ID --azure-rm-service-principal-id $SERVICE_PRIN_ID --name "$SERVICE_CONN_NAME" +``` + +> NOTE: The above command may prompt a request for the 'Azure RM service principal key'. Use the key that was only shown once upon creating the Service Principle from within the Azure Portal. + +#### c. Configure "*Infrastructure Pipeline Variables*" as the first of two variable groups + +Variable groups are utilized by the pipeline to configure how the Cobalt Infrastructure Template will be tested and deployed. The `az pipelines variable-group create` `--variables` flag expects a list of space-delimited key value pairs (e.g., `KEY1='val1' KEY2=true`). + +```bash +# IMPORTANT: Replace these values as necessary to fit your environment. +az pipelines variable-group create --authorize true --name "$COBALT_VAR_GROUP_INFRA" --variables \ + AGENT_POOL="$AGENT_POOL_NAME" \ + ARM_PROVIDER_STRICT=true \ + BUILD_ARTIFACT_NAME='drop' \ + BUILD_ARTIFACT_PATH_ALIAS='artifact' \ + GO_VERSION='1.12.5' \ + PIPELINE_ROOT_DIR='devops/providers/azure-devops/templates/infrastructure' \ + REMOTE_STATE_CONTAINER='' \ + SCRIPTS_DIR='scripts' \ + TEST_HARNESS_DIR='test-harness/' \ + TF_ROOT_DIR='infra' \ + TF_VERSION='0.12.4' \ + TF_WARN_OUTPUT_ERRORS=1 +``` + +| Name | Value | Var Description | +|-------------|-----------|-----------| +| `AGENT_POOL` | `Hosted Ubuntu 1604` | Determines the type of build agent used for deployment. | +| `ARM_PROVIDER_STRICT` | true | Terraform ARM provider modification | +| `BUILD_ARTIFACT_NAME` | drop | Name to identity the folder containing artifacts output by a build. | +| `GO_VERSION`| 1.12.5 | The version of Go terraform deployments are bound to. | +| `PIPELINE_ROOT_DIR` | devops/providers/azure-devops/templates/ | A path for finding Cobalt Infrastructure Templates. | +| `REMOTE_STATE_CONTAINER` | `BACKEND_STATE_CONTAINER_NAME`| The remote blob storage container name for managing the state of a deployed Cobalt Infrastructure Template. Also is used as a naming convention for partitioning state into multiple workspaces. This name was created in an earlier step from within the azure portal. | +| `SCRIPTS_DIR` | infrastructure/scripts | Path to scripts used at runtime for composing build and release jobs at various pipeline stages. | +| `TEST_HARNESS_DIR` | test-harness/ | A path to the cobalt test harness for running integration and unit tests written in Docker and Golang. | +| `TF_ROOT_DIR`| infra | The primary path for all Cobalt Infrastructure Templates and the modules they are composed of. | +| `TF_VERSION`| 0.12.4 | The version of terraform deployments are bound to. | +| `TF_WARN_OUTPUT_ERRORS`| 1 | The severity level of errors to report. | + +#### d. Configure "*DevInt Environment Variables*" as the final variable group + +```bash +# IMPORTANT: Replace these values as necessary to fit your environment. +DEVINT_VAR_GROUP="DevInt $COBALT_VAR_GROUP_ENV_SUFFIX" +az pipelines variable-group create --authorize true --name "$DEVINT_VAR_GROUP" --variables \ + ARM_SUBSCRIPTION_ID='TARGETSUBSCRIPTIONID' \ + REMOTE_STATE_ACCOUNT='BACKENDSTATESTORAGEACCOUNTNAME' \ + SERVICE_CONNECTION_NAME="$SERVICE_CONN_NAME" \ + FORCE_RUN=true +``` + +| Name | Value | Var Description | +|-------------|-----------|-----------| +| `ARM_SUBSCRIPTION_ID` | `` | The Azure subscription ID for which all resources will be deployed. Refer to the Azure subscription chosen in Azure portal for Cobalt deployments. | +| `REMOTE_STATE_ACCOUNT` | `` | The storage container account name created in a previous step that is used to manage the state of this deployment pipeline. The storage Account is shared among all non-prod deployment stages. | +| `SERVICE_CONNECTION_NAME` | ex. Cobalt Deployment Administrator-`` | The custom name of the service connection configured in a previous Azure DevOps step that establishes a connection between the Service Principal and the Azure subscription that it's permissioned for. | + +#### e. Link the variable groups to the build pipeline + +> NOTE: At this time, the Azure DevOps CLI does not support linking variable groups to pipelines. We have a temporary workaround utilizing the Azure DevOps `invoke` command to directly call the Azure DevOps REST API to update the build definition. + +Write the current value of the build pipeline definition to a temporary local file, and save the PIPELINE_ID. + +```bash +az pipelines show --name "$COBALT_PIPELINE_NAME" -o json > builddef.json +PIPELINE_ID=$(az pipelines show --name "$COBALT_PIPELINE_NAME" --query id) +``` + +Execute the list command to find the Variable Group IDs created earlier. Make note of the IDs as they will need to be added to the build pipeline definition. + +```bash +az pipelines variable-group list +``` + +Here, you'll be manually editing the builddef.json file to add the variable group references. At the end of the file, you should see the line `"variableGroups" : null`. Replace the value with the following, replacing the variable group ID placeholders (`0`) with those from the above command for the Infrastructure Pipeline Variables group and DevInt Environment Variables group: + +```bash +"variableGroups": [ + { "id": 0 }, + { "id": 0 } +], +``` + +Save the file. Use the az devops invoke command to update the pipeline build definition with the linked variable groups. + +```bash +az devops invoke --http-method PUT --area build --resource definitions --route-parameters project="$TEMPLATE_DEVOPS_PROJECT_NAME" definitionId=$PIPELINE_ID --in-file builddef.json +``` + +### 5. Keep the Cobalt Infrastructure Templates relevant to your enterprise patterns and run the pipeline + +The goal of this step is to remove Cobalt Infrastructure Templates that owners of the project have no interest in deploying. + +#### a. Clone newly created Azure DevOps Repo from your organization + +Clone the project into a local repo directory + +```bash +git clone +``` + +#### b. Review devops pipeline YAML + +Remove jobName configurations that reference Cobalt Infrastructure Templates not intended for deployment. If new to Cobalt, we recommend keeping the path to the az_hello_world template as a starter template. Below is an example of a jobName that you may want to remove by simply deleting the section from the file. + +```yaml +configurationMatrix: +- jobName: az_service_single_region +terraformTemplatePath: 'infra/templates/az-service-single-region' +terraformWorkspacePrefix: 'sr' +environmentsToTeardownAfterRelease: +- 'devint' +``` + +#### c. Review Cobalt Infrastructure Templates + +Manually delete Cobalt Infrastructure Template folder directories not needed for your enterprise. These will most likely reflect the same template references removed from the devops pipeline YAML. (Do not delete 'backend-state-setup' template! We also recommended keeping the 'az-hello-world' Cobalt Infrastructure Template as a starter template.) + +![image](https://user-images.githubusercontent.com/10041279/64913136-1d1e2f00-d700-11e9-95cd-9e95c257bcbd.png) + +> NOTE: The CI/CD pipeline needs to detect a code change to run the template-specific build and release jobs (in their respective stages). To force the template build and release to run, you may add a `FORCE_RUN` environment variable with a value of `true` to your *Devint Environment Variables* variable group. You may also add a comment or extra line to a TF or Go file within the template in order for the pipeline script to detect a change without adding any additional override flags + +#### b. Commit, Run and verify + +```bash +git commit -m "Removed unrelated templates." && git push +az pipelines run --name "$COBALT_PIPELINE_NAME" +``` + +## Additional Recommendations + +Recommended next step is to target containerized applications via their image names from within a Cobalt Infrastructure Template. diff --git a/docs/GETTING_STARTED_APP_DEV_CLI.md b/docs/GETTING_STARTED_APP_DEV_CLI.md index 0d5f979d..757a0729 100644 --- a/docs/GETTING_STARTED_APP_DEV_CLI.md +++ b/docs/GETTING_STARTED_APP_DEV_CLI.md @@ -4,7 +4,7 @@ ## Overview -This section provides application developers wishing to host solutions on Cobalt templates recommendations for building their infrastructure-as-code repository and accompanying CI/CD pipelines. It assumes an isolated Azure DevOps Project with a Cobalt template Repo and Build Pipeline has already been created as defined in the [Getting Started - Advocated Pattern Owner](./GETTING_STARTED_ADD_PAT_OWNER.md) walkthrough. +This section provides application developers wishing to host solutions on Cobalt templates recommendations for building their infrastructure-as-code repository and accompanying CI/CD pipelines. It assumes an isolated Azure DevOps Project with a Cobalt template Repo and Build Pipeline has already been created as defined in the [Getting Started - Advocated Pattern Owner - Azure CLI](./GETTING_STARTED_ADD_PAT_OWNER_CLI.md) walkthrough. By creating an application-specific project, you are creating a single project supporting the two main pillars of an application -- the Cobalt-template-based infrastructure and CI/CD build pipeline, and the application code and CI/CD build pipeline. **Important**: as an application developer, you will not be modifying the Cobalt template even though you will be importing all of the required code into your project repository. Instead, you will be responsible only for modifying the configuration via the template's `terraform.tfvars` file to support your application's unique settings (e.g., the number of deployment targets to create or Azure Container Registry image URLs). @@ -70,7 +70,7 @@ az devops configure -d project="$APP_DEVOPS_PROJECT_NAME" ### 3. Setup Azure DevOps Repo for Cobalt source -Create a new repository for the Cobalt source within your application project. Import the source from your organizational Cobalt template repository as created in the [Getting Started - Advocated Patterns Owner](./GETTING_STARTED_ADD_PAT_OWNER.md). +Create a new repository for the Cobalt source within your application project. Import the source from your organizational Cobalt template repository as created in the [Getting Started - Advocated Patterns Owner - Azure CLI](./GETTING_STARTED_ADD_PAT_OWNER_CLI.md). ```bash az repos create --name "$APP_DEVOPS_INFRA_REPO_NAME" @@ -118,6 +118,7 @@ The following *Infrastructure Pipeline Variables* are used by all possible envir ```bash # IMPORTANT: Replace these values as necessary to fit your environment. +# IMPORTANT: REMOTE_STATE_CONTAINER should hold a value that is different than the source repo. az pipelines variable-group create --authorize true --name "$COBALT_VAR_GROUP_INFRA" --variables \ AGENT_POOL='Hosted Ubuntu 1604' \ ARM_PROVIDER_STRICT=true \ @@ -141,6 +142,7 @@ Within the pipeline build definition you may specify the number of environments For this walkthrough, we will only create a single environment -- *devint*. The following commands will create the required *DevInt Environment Variables* variable group. ```bash # IMPORTANT: Replace these values as necessary to fit your environment. +# IMPORTANT: If you have control over the service connection, it should rely on the same SP backing the source repo service connection. DEVINT_VAR_GROUP="DevInt $COBALT_VAR_GROUP_ENV_SUFFIX" az pipelines variable-group create --authorize true --name $DEVINT_VAR_GROUP --variables \ ARM_SUBSCRIPTION_ID='TARGETSUBSCRIPTIONID' \ @@ -149,7 +151,7 @@ az pipelines variable-group create --authorize true --name $DEVINT_VAR_GROUP --v TF_CLI_ARGS='' ``` -> NOTE: The Service Connection name should be provided by someone in your organziation with the *Global administrator* permission for your Azure Active Directory tenant. If it has not been provisisioned for you, you may create another by following the directions outlined in the [Getting Started - Advocated Pattern Onwer documentation](./GETTING_STARTED_ADD_PAT_OWNER.md) +> NOTE: The Service Connection name should be provided by someone in your organization with the *Global administrator* permission for your Azure Active Directory tenant. If it has not been provisisioned for you, you may create another by following the directions outlined in the [Getting Started - Advocated Pattern Owner CLI documentation](./GETTING_STARTED_ADD_PAT_OWNER_CLI.md) At this time, the Azure DevOps CLI does not support linking variable groups to pipelines. We have a temporary workaround utilizing the Azure DevOps `invoke` command to directly call the Azure DevOps REST API to update the build definition. @@ -184,7 +186,7 @@ Queue a pipeline to run. az pipelines run --name "$COBALT_PIPELINE_NAME" ``` -Because you have cloned a pipeline definition that was created from the [Getting Started - Advocated Pattern Owner](./GETTING_STARTED_ADD_PAT_OWNER.md) walkthrough, the pipeline definition may be setup to tear down the infrastructure provisioned. For this step in the end-to-end process, we would like the environment to be durable and persist beyond the pipeline execution. Check the primary `azure-pipelines.yml` file's stages. Verify that the `configurationMatrix` does not include an `environmentsToTeardownAfterRelease` property. If it does, remove it so that the environment remains available for use by the application after the pipeline succeeds. +Because you have cloned a pipeline definition that was created from the [Getting Started - Advocated Pattern Owner CLI](./GETTING_STARTED_ADD_PAT_OWNER_CLI.md) walkthrough, the pipeline definition may be setup to tear down the infrastructure provisioned. For this step in the end-to-end process, we would like the environment to be durable and persist beyond the pipeline execution. Check the primary `azure-pipelines.yml` file's stages. Verify that the `configurationMatrix` does not include an `environmentsToTeardownAfterRelease` property. If it does, remove it so that the environment remains available for use by the application after the pipeline succeeds. To host your application on this provisioned environment, update the `*.tfvars` file specific to your template to ensure your application is being deployed to the infrastructure. You may also need to add values to your provisioned Azure Key Vault resource for the application to work as expected. diff --git a/docs/images/AppReg.png b/docs/images/AppReg.png new file mode 100644 index 00000000..c95d797e Binary files /dev/null and b/docs/images/AppReg.png differ diff --git a/docs/images/Org.png b/docs/images/Org.png new file mode 100644 index 00000000..9eaf52c1 Binary files /dev/null and b/docs/images/Org.png differ diff --git a/docs/images/api-permissions.png b/docs/images/api-permissions.png new file mode 100644 index 00000000..6a483498 Binary files /dev/null and b/docs/images/api-permissions.png differ diff --git a/docs/images/graph.png b/docs/images/graph.png new file mode 100644 index 00000000..4a06f309 Binary files /dev/null and b/docs/images/graph.png differ diff --git a/docs/images/pipline.gif b/docs/images/pipline.gif new file mode 100644 index 00000000..f53eb177 Binary files /dev/null and b/docs/images/pipline.gif differ diff --git a/docs/images/pipline.png b/docs/images/pipline.png new file mode 100644 index 00000000..aa421838 Binary files /dev/null and b/docs/images/pipline.png differ diff --git a/docs/images/preview.png b/docs/images/preview.png new file mode 100644 index 00000000..6ef71d1b Binary files /dev/null and b/docs/images/preview.png differ diff --git a/docs/images/project.png b/docs/images/project.png new file mode 100644 index 00000000..c3dd429f Binary files /dev/null and b/docs/images/project.png differ