CICD Pipeline Module: Databricks

[jose-perales]

Description

Our goal is to create a Databricks CICD Pipeline module, that takes the best practices of all our existing fabric samples and provides engineers a fast path to set up a CICD Pipeline for data engineering work. We want to take into account:

• Optionality on permissions (Entra Global Admin vs Azure Contributor)
• Support both first party native integrations and git native intergrations, if possible
• Composable on top of our Databricks Infra Module

Our audience/persona are software and data engineers hoping to easily set up CICD Pipelines while working with Databricks.

Acceptance Criteria

• Must use a devcontainer
• Spike on AZD CLI
• Spike on Deployment Stacks
• Testing approach documented