Add Azure Modular Recipe/Stack Deployment Option

[strickvl]

While MLStacks currently supports GCP and AWS for stack deployments, there is a notable absence of support for Azure. Although the creation of remote state on Azure is facilitated through a Terraform registry module implemented by ZenML), comprehensive support for Azure stack deployments through modular Terraform recipes is missing. This task aims to bridge that gap by introducing Azure as a fully supported provider within MLStacks.

Task Description

The objective is to develop a series of Terraform modules for Azure, analogous to those available for AWS and GCP within MLStacks. This will enable users to deploy selective parts of their MLStacks on Azure. The task involves leveraging existing Terraform code (e.g., azure-minimal and azureml-minimal) and adapting it to the modular approach used in MLStacks. Full testing is required to ensure the Azure implementation's functionality and compatibility with MLStacks.

Expected Outcome

• Azure is integrated as a fully supported provider within MLStacks, with modular Terraform recipes allowing for selective deployment of stack components.
• Users can deploy MLStacks components on Azure, similar to the existing support for GCP and AWS.
• Documentation is updated to guide users on deploying MLStacks on Azure, including setup, configuration, and usage.
• Comprehensive testing verifies the Azure deployment option's functionality, ensuring compatibility and reliability within the MLStacks framework.

Steps to Implement

1. Review existing Azure-related Terraform code within MLStacks and related repositories to identify reusable components.
2. Develop modular Terraform recipes for Azure, ensuring they align with the structure and approach used for AWS and GCP within MLStacks.
3. Implement Azure stack components within MLStacks, ensuring they support the dynamic selection and deployment of various stack parts.
4. Conduct thorough testing of the Azure deployment option across different configurations and scenarios to ensure full functionality and integration within MLStacks.
5. Update the MLStacks documentation to include comprehensive guidance on using the Azure deployment option, covering the setup, configuration, deployment, and best practices.
6. Engage with the MLStacks community to gather feedback on the Azure integration and make adjustments as necessary to address any issues or enhancements.

Additional Context

Expanding MLStacks to include Azure as a supported provider will significantly enhance the framework's versatility and appeal to users operating within the Azure ecosystem. This addition is a strategic move to ensure MLStacks remains a comprehensive and flexible solution for MLOps infrastructure deployment across all major cloud providers.

Code of Conduct

• I agree to follow this project's Code of Conduct

[MASisserson]

I'd like to take a stab at this, if that's alright.

[strickvl]

Sure thing! Let me know if you have questions along the way. I'd recommend starting small (just artifact store to start with) and then slowly adding new components incrementally. You might also draw some inspiration from these non-modular recipes from the 'old' mlstacks: https://github.com/zenml-io/mlstacks/tree/main/azure-minimal and https://github.com/zenml-io/mlstacks/tree/main/azureml-minimal

[MASisserson]

Thanks for the suggestions!

[MASisserson]

I'm sorry. I can't accomplish this one. I'm in too far over my head with it and can't make meaningful progress.

[strickvl]

No worries @MASisserson! Thanks for giving it a shot. We're also working on a parallel effort around Terraform recipes which might be backported into mlstacks at some point...