diff --git a/README.md b/README.md
index 29c237b..f29320a 100644
--- a/README.md
+++ b/README.md
@@ -16,7 +16,7 @@ urlFragment: "jmeter-aci-terraform"
# Load Testing Pipeline with JMeter, ACI and Terraform
-This project is a load testing pipeline that leverages [Apache JMeter](https://jmeter.apache.org/) as an open source load and performance testing tool and [Terraform](https://www.terraform.io/) to dynamically provision and destroy the required infrastructure on Azure.
+This project is a load testing pipeline that leverages [Apache JMeter](https://jmeter.apache.org/) as an open source load and performance testing tool and [Terraform](https://www.terraform.io/) to dynamically provision and destroy the required infrastructure on Azure.
## Key concepts
@@ -26,7 +26,7 @@ The flow is triggered and controlled by an [Azure Pipeline](https://azure.micros
| Task group | Tasks |
|-------------------------|--------|
-| SETUP |
Check if the JMeter Docker image existsValidate the JMX file that contains the JMeter test definitionUpload JMeter JMX file to Azure Storage Account File ShareProvision the infrastructure with Terraform |
+| SETUP | Retrieve secrets from Azure Key VaultCheck if the JMeter Docker image existsValidate the JMX file that contains the JMeter test definitionUpload JMeter JMX file to Azure Storage Account File ShareProvision the infrastructure with Terraform |
| TEST | Run JMeter test execution and wait for completion |
| RESULTS | Show JMeter logsGet JMeter artifacts (e.g. logs, dashboard)Convert JMeter tests result (JTL format) to JUnit formatPublish JUnit test results to Azure PipelinesPublish JMeter artifacts to Azure Pipelines |
| TEARDOWN | Destroy all ephemeral infrastructure with Terraform |
@@ -60,6 +60,8 @@ On the `RESULTS` phase, a [JMeter Report Dashboard](https://jmeter.apache.org/us
* [Azure DevOps CLI](https://docs.microsoft.com/en-us/azure/devops/cli/?view=azure-devops)
* [Service Principal](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli?view=azure-cli-latest)
* [Azure Container Registry](https://azure.microsoft.com/en-us/services/container-registry/)
+* [Azure Service Connection](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml)
+* [Azure Key Vault](https://azure.microsoft.com/en-us/services/key-vault/)
* Shell
* [jq](https://stedolan.github.io/jq/download/)
@@ -93,11 +95,11 @@ az repos import create --git-source-url $REPOSITORY_URL --repository $REPOSITORY
### 2. Creating or reusing a service principal
-Azure service principal is an identity created for use with applications, hosted services, and automated tools to access Azure resources. This access is restricted by the roles assigned to the service principal, giving you control over which resources can be accessed and at which level.
+Azure service principal is an identity created for use with applications, hosted services, and automated tools to access Azure resources. This access is restricted by the roles assigned to the service principal, giving you control over which resources can be accessed and at which level.
Terraform requires a service principal to authenticate to Azure. You can use an existing service principal or create a new one through Azure CLI or Azure Portal.
-You can follow the steps described [here](https://www.terraform.io/docs/providers/azurerm/guides/service_principal_client_secret.html#creating-a-service-principal-using-the-azure-cli) to create a service principal using the Azure CLI. Make sure you copied the `appId`, `password` and `tenant` properties. In the next steps, they will be used as `CLIENT_ID`, `CLIENT_SECRET` and `TENANT_ID`, respectively.
+You can follow the steps described [here](https://www.terraform.io/docs/providers/azurerm/guides/service_principal_client_secret.html#creating-a-service-principal-using-the-azure-cli) to create a service principal using the Azure CLI. Make sure you copied the `appId`, `password` and `tenant` properties. In the next steps, they will be used as `CLIENT_ID`, `CLIENT_SECRET` and `TENANT_ID`, respectively.
### 3. Getting the subscription ID
@@ -123,48 +125,67 @@ It is expected to get a similar response:
Then copy the `id` property value. It will be used in the next step as `SUBSCRIPTION_ID`.
-### 4. Create Variable Groups
+### 4. Create Azure Devops Service Connection
-Get you service principal, your ACR credentials, and fill the following empty variables. Then, run this block on Bash:
+Fill the following empty variables below and run this block on Bash, there is a prompt for `CLIENT_SECRET` during service connection creation.
+You can assign any service connection name.
```shell
CLIENT_ID=
-CLIENT_SECRET=
-TENANT_ID=
SUBSCRIPTION_ID=
-ACR_NAME=
-ACR_PASSWORD=
+TENANT_ID=
+SUBSCRIPTION_NAME=
+SERVICE_CONNECTION_NAME=
+
+az devops service-endpoint azurerm create --azure-rm-service-principal-id ${CLIENT_ID} \
+ --azure-rm-subscription-id ${SUBSCRIPTION_ID} --azure-rm-subscription-name ${SUBSCRIPTION_NAME} \
+ --azure-rm-tenant-id ${TENANT_ID} --name ${SERVICE_CONNECTION_NAME}
```
-> Note: Make sure the `ACR_NAME` doesn't contain any capital letter, as it's an invalid ACR name convention.
+> NOTE: If a mistake is made in the data provided above, for eg. incorrect service principle ID,
+>the service connection will still be created. You should always manually verify the connection using Azure Devops
+>project settings after creation.
+
+### 5. Create Variable Groups
+Ensure below steps are complete before creating variable groups on Azure Devops:
+- Azure Service Connection name from the step above
+- Azure key vault name
+- Access policy in Azure key vault to allow Azure service principal used to set up service connection, access to get keys from the vault
+- Set up secrets within the key vault with keys specified below:
+ - SUBSCRIPTION_ID with key arm-subscription-id
+ - TENANT_ID with key arm-tenant-id
+ - CLIENT_ID with key arm-client-id
+ - CLIENT_SECRET with key arm-client-secret
+ - Azure Container Registry password with key acr-secret
-Then run the following commands to create the variable groups `JMETER_AZURE_PRINCIPAL` and `JMETER_TERRAFORM_SETTINGS`:
+Now fill the following empty variables below and run this block on Bash:
```shell
-PRIN_GROUP_ID=$(az pipelines variable-group create --name JMETER_AZURE_PRINCIPAL --authorize \
- --variables ARM_CLIENT_ID=$CLIENT_ID \
- ARM_TENANT_ID=$TENANT_ID \
- ARM_SUBSCRIPTION_ID=$SUBSCRIPTION_ID \
- | jq .id)
+ACR_NAME=
+KEY_VAULT_NAME=
+SERVICE_CONNECTION_NAME=
+```
+
+> Note: Make sure the `ACR_NAME` doesn't contain any capital letter, as it's an invalid ACR name convention.
+
+Then run the following commands to create the variable group `JMETER_TERRAFORM_SETTINGS`:
-az pipelines variable-group variable create --group-id $PRIN_GROUP_ID --secret true \
- --name ARM_CLIENT_SECRET \
- --value $CLIENT_SECRET
+```shell
SETT_GROUP_ID=$(az pipelines variable-group create --name JMETER_TERRAFORM_SETTINGS --authorize \
--variables TF_VAR_JMETER_IMAGE_REGISTRY_NAME=$ACR_NAME \
TF_VAR_JMETER_IMAGE_REGISTRY_USERNAME=$ACR_NAME \
TF_VAR_JMETER_IMAGE_REGISTRY_SERVER=$ACR_NAME.azurecr.io \
+ TF_VAR_SERVICE_CONNECTION_NAME="$SERVICE_CONNECTION_NAME" \
+ TF_VAR_KEY_VAULT_NAME=$KEY_VAULT_NAME \
TF_VAR_JMETER_DOCKER_IMAGE=$ACR_NAME.azurecr.io/jmeter \
| jq .id)
-az pipelines variable-group variable create --group-id $SETT_GROUP_ID --secret true \
- --name TF_VAR_JMETER_IMAGE_REGISTRY_PASSWORD \
- --value $ACR_PASSWORD
+az pipelines variable-group variable create --group-id $SETT_GROUP_ID
```
-### 5. Create and Run the Docker Pipeline
+### 6. Create and Run the Docker Pipeline
```shell
PIPELINE_NAME_DOCKER=jmeter-docker-build
@@ -174,7 +195,7 @@ az pipelines create --name $PIPELINE_NAME_DOCKER --repository $REPOSITORY_NAME \
--yml-path pipelines/azure-pipelines.docker.yml
```
-### 6. Create the JMeter Pipeline
+### 7. Create the JMeter Pipeline
```shell
PIPELINE_NAME_JMETER=jmeter-load-test
@@ -187,11 +208,11 @@ az pipelines variable create --pipeline-name $PIPELINE_NAME_JMETER --name TF_VAR
az pipelines variable create --pipeline-name $PIPELINE_NAME_JMETER --name TF_VAR_JMETER_WORKERS_COUNT --allow-override
```
-### 7. Update the JMX test definition (optional)
+### 8. Update the JMX test definition (optional)
By default, this repository uses a `sample.jmx` file under the `jmeter` folder. This JMX file contains a test definition for performing HTTP requests on `azure.microsoft.com` endpoint through the `443` port. You can simply update the it with the test definition of your preference.
-### 8. Manually Run the JMeter Pipeline
+### 9. Manually Run the JMeter Pipeline
You can choose the JMeter file you want to run (e.g. [jmeter/sample.jmx](./jmeter/sample.jmx)) and how many JMeter workers you will need for your test. Then you can run the JMeter pipeline using the CLI:
diff --git a/pipelines/azure-pipelines.docker.yml b/pipelines/azure-pipelines.docker.yml
index 7b2eb1c..06be4f2 100644
--- a/pipelines/azure-pipelines.docker.yml
+++ b/pipelines/azure-pipelines.docker.yml
@@ -10,15 +10,15 @@ pool:
vmImage: 'ubuntu-latest'
variables:
-- group: JMETER_AZURE_PRINCIPAL
- group: JMETER_TERRAFORM_SETTINGS
steps:
-- script: |
- az login --service-principal --username $(ARM_CLIENT_ID) --password $(ARM_CLIENT_SECRET) --tenant $(ARM_TENANT_ID)
- az account set --subscription $(ARM_SUBSCRIPTION_ID)
- displayName: 'Login on Azure CLI'
-- script: |
- az acr build -t $(TF_VAR_JMETER_DOCKER_IMAGE) -r $(TF_VAR_JMETER_IMAGE_REGISTRY_NAME) -f $(Build.SourcesDirectory)/docker/Dockerfile .
+- task: AzureCLI@2
displayName: 'Build and Push JMeter Docker image'
+ inputs:
+ azureSubscription: $(TF_VAR_SERVICE_CONNECTION_NAME)
+ scriptType: bash
+ scriptLocation: inlineScript
+ inlineScript: |
+ az acr build -t $(TF_VAR_JMETER_DOCKER_IMAGE) -r $(TF_VAR_JMETER_IMAGE_REGISTRY_NAME) -f $(Build.SourcesDirectory)/docker/Dockerfile $(Build.SourcesDirectory)/docker
\ No newline at end of file
diff --git a/pipelines/azure-pipelines.load-test.yml b/pipelines/azure-pipelines.load-test.yml
index 8baffbb..241efbd 100644
--- a/pipelines/azure-pipelines.load-test.yml
+++ b/pipelines/azure-pipelines.load-test.yml
@@ -4,7 +4,6 @@ pool:
vmImage: 'ubuntu-18.04'
variables:
-- group: JMETER_AZURE_PRINCIPAL
- group: JMETER_TERRAFORM_SETTINGS
- name: JMETER_DIRECTORY_INPUT
value: $(System.DefaultWorkingDirectory)/jmeter
@@ -13,15 +12,22 @@ variables:
steps:
-- script: |
- az login --service-principal --username $(ARM_CLIENT_ID) --password $(ARM_CLIENT_SECRET) --tenant $(ARM_TENANT_ID)
- az account set --subscription $(ARM_SUBSCRIPTION_ID)
- displayName: 'SETUP: Login on Azure CLI'
+- task: AzureKeyVault@1
+ inputs:
+ azureSubscription: $(TF_VAR_SERVICE_CONNECTION_NAME)
+ KeyVaultName: $(TF_VAR_KEY_VAULT_NAME)
+ SecretsFilter: '*'
+ displayName: 'Get secrets from Azure Key Vault'
-- script: |
- az acr login -n $(TF_VAR_JMETER_IMAGE_REGISTRY_NAME)
- docker pull $(TF_VAR_JMETER_DOCKER_IMAGE)
+- task: AzureCLI@2
displayName: 'SETUP: Validate JMeter Docker Image'
+ inputs:
+ azureSubscription: $(TF_VAR_SERVICE_CONNECTION_NAME)
+ scriptType: bash
+ scriptLocation: inlineScript
+ inlineScript: |
+ az acr login -n $(TF_VAR_JMETER_IMAGE_REGISTRY_NAME)
+ docker pull $(TF_VAR_JMETER_DOCKER_IMAGE)
- script: |
docker run --name=jmx-validator -v $(JMETER_DIRECTORY_INPUT):/jmeter -w /jmeter \
@@ -36,8 +42,11 @@ steps:
- script: terraform apply -target azurerm_storage_share.jmeter_share -auto-approve
workingDirectory: ./terraform
env:
- ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
- TF_VAR_JMETER_IMAGE_REGISTRY_PASSWORD: $(TF_VAR_JMETER_IMAGE_REGISTRY_PASSWORD)
+ ARM_SUBSCRIPTION_ID: $(arm-subscription-id)
+ ARM_TENANT_ID: $(arm-tenant-id)
+ ARM_CLIENT_ID: $(arm-client-id)
+ ARM_CLIENT_SECRET: $(arm-client-secret)
+ TF_VAR_JMETER_IMAGE_REGISTRY_PASSWORD: $(acr-secret)
displayName: 'SETUP: Run Terraform Apply (target=file share)'
- script: |
@@ -52,39 +61,54 @@ steps:
- script: terraform apply -auto-approve
workingDirectory: ./terraform
env:
- ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
- TF_VAR_JMETER_IMAGE_REGISTRY_PASSWORD: $(TF_VAR_JMETER_IMAGE_REGISTRY_PASSWORD)
+ ARM_SUBSCRIPTION_ID: $(arm-subscription-id)
+ ARM_TENANT_ID: $(arm-tenant-id)
+ ARM_CLIENT_ID: $(arm-client-id)
+ ARM_CLIENT_SECRET: $(arm-client-secret)
+ TF_VAR_JMETER_IMAGE_REGISTRY_PASSWORD: $(acr-secret)
displayName: 'SETUP: Run Terraform Apply (target=all)'
-- script: |
- RG=$(terraform output resource_group_name)
- NAME=$(terraform output jmeter_controller_name)
- echo "`date`: Started!"
- while [ $(az container show -g $RG -n $NAME --query "containers[0].instanceView.currentState.state" -o tsv) == "Running" ]; do
- echo "`date`: Still Running..."
- sleep 20
- done
- echo "`date`: Finished!"
- workingDirectory: ./terraform
+- task: AzureCLI@2
+ inputs:
+ azureSubscription: $(TF_VAR_SERVICE_CONNECTION_NAME)
+ workingDirectory: ./terraform
+ scriptType: bash
+ scriptLocation: inlineScript
+ inlineScript: |
+ RG=$(terraform output resource_group_name)
+ NAME=$(terraform output jmeter_controller_name)
+ echo "`date`: Started!"
+ while [ $(az container show -g $RG -n $NAME --query "containers[0].instanceView.currentState.state" -o tsv) == "Running" ]; do
+ echo "`date`: Still Running..."
+ sleep 20
+ done
+ echo "`date`: Finished!"
displayName: 'TEST: Wait Test Execution'
-- script: az container logs -g $(terraform output resource_group_name) -n $(terraform output jmeter_controller_name)
- workingDirectory: ./terraform
- displayName: 'RESULTS: Show JMeter Controller Logs'
-
-- script: |
- RESOURCE_GROUP=$(terraform output resource_group_name)
- echo -n $(terraform output jmeter_workers_names) | xargs -t -d "," -I '{}' -n1 az container logs -g $RESOURCE_GROUP -n {}
- workingDirectory: ./terraform
- displayName: 'RESULTS: Show JMeter Worker Logs'
-
-- script: |
- azcopy \
- --source $(terraform output storage_file_share_url) \
- --destination $(JMETER_DIRECTORY_OUTPUT) \
- --source-key $(terraform output storage_account_key) \
- --recursive
- workingDirectory: ./terraform
+- task: AzureCLI@2
+ inputs:
+ azureSubscription: $(TF_VAR_SERVICE_CONNECTION_NAME)
+ workingDirectory: ./terraform
+ scriptType: bash
+ scriptLocation: inlineScript
+ inlineScript: |
+ az container logs -g $(terraform output resource_group_name) -n $(terraform output jmeter_controller_name)
+ RESOURCE_GROUP=$(terraform output resource_group_name)
+ echo -n $(terraform output jmeter_workers_names) | xargs -t -d "," -I '{}' -n1 az container logs -g $RESOURCE_GROUP -n {}
+ displayName: 'RESULTS: Collect JMeter Controller and Worker Logs'
+
+- task: AzureCLI@2
+ inputs:
+ azureSubscription: $(TF_VAR_SERVICE_CONNECTION_NAME)
+ workingDirectory: ./terraform
+ scriptType: bash
+ scriptLocation: inlineScript
+ inlineScript: |
+ azcopy \
+ --source $(terraform output storage_file_share_url) \
+ --destination $(JMETER_DIRECTORY_OUTPUT) \
+ --source-key $(terraform output storage_account_key) \
+ --recursive
displayName: 'RESULTS: Get JMeter Artifacts'
- script: |
@@ -108,8 +132,10 @@ steps:
- script: terraform destroy -auto-approve
workingDirectory: ./terraform
- condition: succeededOrFailed()
env:
- ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
- TF_VAR_JMETER_IMAGE_REGISTRY_PASSWORD: $(TF_VAR_JMETER_IMAGE_REGISTRY_PASSWORD)
- displayName: 'TEARDOWN: Run Terraform Destroy'
+ ARM_SUBSCRIPTION_ID: $(arm-subscription-id)
+ ARM_TENANT_ID: $(arm-tenant-id)
+ ARM_CLIENT_ID: $(arm-client-id)
+ ARM_CLIENT_SECRET: $(arm-client-secret)
+ TF_VAR_JMETER_IMAGE_REGISTRY_PASSWORD: $(acr-secret)
+ displayName: 'TEARDOWN: Run Terraform Destroy'
\ No newline at end of file