Infra Module: Databricks

[jose-perales]

Description

Our goal is to create a Databricks platform infrastructure deployment module, that takes the best practices of all our existing databricks samples and simplifies the infra deployment. We want to take into account:

• Optionality on permissions (Entra Global Admin vs Azure Contributor)
• Storage Accounts, Security Groups if necessary
• Service Principal vs Managed Identity permissioning
  • Needs to be documented (see parking sensors sample for examples on each)

Our audience/persona are software and data engineers hoping to easily deploy the necessary compute and storage to set up a platform Databricks deployment.

Acceptance Criteria

• Must use a devcontainer
• Spike on AZD CLI
• Spike on Deployment Stacks
• IaC in Bicep or Terraform (consider using AVMs if they exist)
• Testing approach documented