-
Notifications
You must be signed in to change notification settings - Fork 10
Closed
Description
Summary
Replace the self-hosted GCP VM runners with Modal serverless compute for CI/CD workflows.
Motivation
- Cost reduction: Pay-per-use vs always-on VM (~$180/month → ~$60-80/month)
- No infrastructure management: No VM maintenance, spot instance handling
- Parallelization potential: Local area builds can run 435 CDs concurrently
Changes Required
- Create Modal app with
data_build.pyfor dataset builds - Create Modal app with
local_area.pyfor local area publishing - Update
reusable_test.yamlto trigger Modal instead of self-hosted - Update
local_area_publish.yamlto trigger Modal - Add Modal secrets to GitHub (
MODAL_TOKEN_ID,MODAL_TOKEN_SECRET) - Create Modal secrets (
huggingface-token,gcp-credentials) - Validate in production
- Decommission GCP self-hosted runner VM
Testing
Successfully tested modal run modal_app/data_build.py --test-lite --no-upload locally.
Metadata
Metadata
Assignees
Labels
No labels