Sample Python KEDA Service Bus Scaler

Sample Code for Python based Kubernetes Jobs for Event-driven Autoscaling using KEDA Azure Service Bus Scaler.

Prerequisite

1. Azure Subscription
2. AKS Cluster
3. Azure Service Bus
4. Python 3
5. Kubernetes CLI (kubectl)

Setup

1. Create an Azure Service Bus Queue

2. Deploy KEDA 2.0+ in AKS Cluster

3. Get the Azure Service Bus connection string

4. Get base64 encoded values of -

   1. Connection String: echo -n "<connection string>" | base64
   2. Queue Name: echo -n "<queue name>" | base64

5. Modify the following configuration files -

   1. keda-config/jobs/secret.yaml: update servicebus-queue and servicebus-connectionstring by the values from above commands

      data:
          servicebus-queue: <base64_encoded_servicebus_queue_name>
          servicebus-connectionstring: <base64_encoded_servicebus_connection_string>

   2. keda-config/jobs/keda.yaml: update servicebus_queue_name and servicebus_namespace with plain text values of respective queue name and namespace name

        triggers:
        - type: azure-servicebus
          metadata:
            # Required: queueName OR topicName and subscriptionName
            queueName: <servicebus_queue_name>
            # Required: Define what Azure Service Bus to authenticate to with Managed Identity
            namespace: <servicebus_namespace>

6. Deploy the KEDA enabled Sample Kubernets Job (in default namespace)

kubectl apply -f keda-config/jobs/secret.yaml
kubectl apply -f keda-config/jobs/keda.yaml

1. Verify deployment

~/sample-python-keda-service-bus-scaler(main*) » kubectl get ScaledJob
NAME                             MAX   TRIGGERS           AUTHENTICATION                READY   ACTIVE   AGE
servicebus-queue-so-sample-job   100   azure-servicebus   auth-service-bus-sample-job                    1s
--------------------------------------------------------------------------------------------------------------
~/sample-python-keda-service-bus-scaler(main*) » kubectl get TriggerAuthentication
NAME                          PODIDENTITY   SECRET              ENV
auth-service-bus-sample-job                 sample-job-secret

Test

1. Populate message in Azure Service Bus Queue

   1. Modify the following configuration file -

      1. test-message-creator/azure-service-bus.yaml: update servicebus_queue_name and servicebus_connection_string with plain text values of respective queue name and connection string

   2. Execute -

      pip3 install -r requirements.txt
      cd test-message-creator && python3 send_message_queue.py

   3. Expected result -

   [07/Jun/2021 15:57:45] INFO - Start sending messages
   .........
   [07/Jun/2021 15:57:53] INFO - Sent a list of 1000 messages
   .........
   [07/Jun/2021 15:57:53] INFO - Done sending messages
   [07/Jun/2021 15:57:53] INFO - -----------------------

2. Monitor AKS Cluster and Azure Service Bus Queue

   1. Monitor AKS Cluster and check if the Job (100) got created as soon as there are messages populated in Azure Service Bus Queue

      ~/sample-python-keda-service-bus-scaler(main*) » kubectl get pods --selector=app=sample-keda-job | grep -v '^NAME' | wc -l
          100
      --------------------------------------------------------------------------------------------------------------
      ~/sample-python-keda-service-bus-scaler(main*) » kubectl get pods
      NAME                                         READY   STATUS      RESTARTS   AGE
      servicebus-queue-so-sample-job-26htm-hw7cm   0/1     Completed   0          2m21s
      servicebus-queue-so-sample-job-26pk7-8sw8g   0/1     Completed   0          2m52s
      ........
      servicebus-queue-so-sample-job-6jp7k-q9tv7   0/1     Completed   0          2m41s
      servicebus-queue-so-sample-job-6qrjg-pfx5n   0/1     Completed   0          2m47s
      --------------------------------------------------------------------------------------------------------------
      ~/sample-python-keda-service-bus-scaler(main*) » kubectl logs servicebus-queue-so-sample-job-xxfvj-l4t86
      started receiving message
      =========================
      Message in list 960
      Message in list 961
      Message in list 962
      Message in list 963
      Message in list 964
      Message in list 965
      Message in list 966
      Message in list 967
      Message in list 968
      Message in list 969
      =========================
      completed receiving message

   2. Check for Messages in Azure Service Bus Respective Queue

      1. Message Count
      2. Message Incoming / Outgoing

(Optional) Build

1. The existing source is built and uploaded to registry - docker.pkg.github.com/prabdeb/sample-python-keda-service-bus-scaler/consumer:latest
2. However if there is any modifications in source code, the docker image can be build by command - docker build -t sample-python-keda-service-bus-scaler:latest -f Dockerfile-job .

Cleaning up resources

1. Delete sample application

   kubectl delete -f keda-config/jobs/keda.yaml
   kubectl delete -f keda-config/jobs/secret.yaml

2. Uninstall KEDA 2.0+

3. Delete Azure Resources