Pipelines
A pipeline is a series of tasks which are executed in a spcific order and during which the result of one task is passed on to the next.
It can be seen like this:
graph TD
A[Task 1] --> |output of task1|B[Task 2]
B --> |output of task2|C[Task 3]
C --> |output of task3|D[Task 4]
D --> |output of task4|E[End]
As you can see, implementing pipelines is a way of adding abstraction to your infrastructure.
Pipelines are a great way to introduce automation to your infrastructure. the specific "flavour" of pipelines that i'm using in this project is event-based pipelines.
I have a little terraform script that deploys a VM on a ProxMox cluster.
This script should run on the least loaded ProxMox node.
So, i write a little python script that checks the load of the ProxMox nodes, and then returns the IP of the least loaded node.
I now have a pipeline that looks like this:
graph TD
A[Check ProxMox node load] --> |IP of least loaded node|B[Deploy VM on ProxMox]
B --> |VM IP|C[Configure VM]
C --> |VM IP|D[End]
This pipeline is a simple example of how you can use pipelines to automate your infrastructure.
In this pipeline the first "task" (the python script) generates an artifact (the IP of the least loaded node) that is then used by the second job (the terraform script) to deploy the VM on the least loaded node.
We can add tasks before or after the the current tasks to add more functionality to the pipeline.
Written by: Louis Declerck
