adoLogStreamer

Overview

Azure DevOps currently offers no out-of-the-box functionality to stream pipeline logs to external endpoints and this sample can be used to fill in the gaps. This project streams ADO pipeline logs to Azure blob storage.

Log Stream Flow

1. Templates/streamLogs.yml template stage gets the pipeline definition run's current context and puts it on the storage queue

2. Function app triggers and processes message

3. Function app calls the Azure DevOps Pipeline List Logs API and downloads the pipeline run's logs

4. Function app pushes the pipeline run's logs to blob storage

Implementation

1. Create or get your Azure DevOps Personal Access Token (PAT)

2. Create a secret in Azure Key Vault named "adoLogStreamerPat" that will securely store the Azure DevOps PAT token. This is set as a keyvault reference in the function app's app settings in the arm template.

$pat = "<pat_token>"
$vaultName = "<keyvault_name>"
Set-AzKeyVaultSecret -VaultName $vaultName -Name "adoLogStreamerPat" -SecretValue $(ConvertTo-SecureString -AsPlainText $pat -Force)

3. Update the pipeline's variables listed in the below table with your own values

   Variable Name	Value Description
   azureSubscription	The name of the service connection the pipeline will leverage to deploy the arm template
   resourceGroupName	The name of the resource group the function app and storage account will be deployed
   location	The region the Azure resource group and resources will be deployed
   functionAppName	The name of the Azure function app. Note this must be a globally unique resource name
   azureDevOpsOrg	The name of the Azure DevOps Organization
   azureDevOpsProject	The name of the Azure DevOps Project
   adoPatKeyvaultSecretUri	The uri of the secret created from #2 - https://{vaultName}.vault.azure.net/secrets/adoLogStreamerPat/
   storageAccountName	The name of the storage account used by the function app to store logs. Note this must be a globally unique resource name

4. Run the pipeline to deploy the arm template and the function app package

5. Grant the function app's MSI access to the keyvault so it can access the adoPatKeyvaultSecretUri keyvault reference app setting

$functionAppName = "<functionApp_name>"
$vaultName = "<keyvault_name>"
$vaultRG = "<keyvault_resourceGroup>"
$msi = Get-AzADServicePrincipal -SearchString $functionAppName
Set-AzKeyVaultAccessPolicy -VaultName $vaultName -ResourceGroupName $vaultRG -ServicePrincipalName $msi.ApplicationId.Guid -PermissionsToSecrets "get, list"