SageMaker ML Pipeline Template

Full native AWS SageMaker ML Pipeline template for end-to-end machine learning workflows.

🎯 Overview

This template provides a complete SageMaker-native ML pipeline that leverages all AWS managed services for maximum scalability and minimum operational overhead.

🔥 What makes this different?

🆚 vs Batch ML Pipeline Template:

• 100% SageMaker native - no custom Docker containers needed
• Fully managed - AWS handles infrastructure, scaling, monitoring
• Integrated Model Registry - automatic model versioning and approval workflows
• Built-in monitoring - SageMaker Model Monitor for data drift detection
• Cost optimized - automatic scaling and spot instances support

🏗️ Architecture

📊 Data Input (S3) 
    ↓
🔄 SageMaker Processing (Feature Engineering)
    ↓  
🤖 SageMaker Training (Model Training)
    ↓
📈 SageMaker Processing (Model Evaluation)
    ↓
📦 SageMaker Model Registry (Conditional Registration)
    ↓
🚀 SageMaker Endpoints (Real-time Inference)

🧩 Components

• SageMaker Pipeline: Orchestrates the entire ML workflow
• SageMaker Processing: Data preprocessing and model evaluation
• SageMaker Training: Distributed model training
• SageMaker Model Registry: Model versioning and governance
• SageMaker Endpoints: Real-time model serving
• SageMaker Model Monitor: Data drift and model quality monitoring

🚀 Quick Start

1. Infrastructure Setup

# Deploy infrastructure
make deploy-infra ENV=dev

# Check AWS configuration
make check-aws

2. Upload Sample Data

# Create sample dataset (replace with your data)
mkdir -p data
echo "feature1,feature2,feature3,target" > data/sample_dataset.csv
echo "1.0,2.0,3.0,0" >> data/sample_dataset.csv
echo "2.0,3.0,4.0,1" >> data/sample_dataset.csv

# Upload to S3
make upload-data ENV=dev

3. Run Pipeline

# Execute the full ML pipeline
make run-pipeline ENV=dev

# Monitor progress
make list-executions ENV=dev

📁 Project Structure

sagemaker-ml-pipeline/
├── configs/                 # Environment configurations
│   ├── dev/
│   ├── staging/
│   └── prod/
├── src/
│   ├── pipeline/           # SageMaker Pipeline definitions
│   │   └── sagemaker_pipeline.py
│   ├── preprocessing/      # Data preprocessing scripts
│   │   └── preprocess.py
│   ├── training/          # Model training scripts
│   │   └── train.py
│   ├── evaluation/        # Model evaluation scripts
│   │   └── evaluate.py
│   └── inference/         # Inference and endpoint management
│       ├── inference.py
│       └── deploy_endpoint.py
├── infrastructure/        # Terraform IaC
│   └── terraform/
│       ├── modules/
│       └── environments/
├── scripts/              # Utility scripts
│   └── run_pipeline.py
├── tests/               # Test suite
└── notebooks/           # Jupyter notebooks

🔧 Configuration

Environment Variables

Create a .env file for local development:

AWS_REGION=us-east-1
AWS_PROFILE=default
SAGEMAKER_ROLE_ARN=arn:aws:iam::ACCOUNT:role/SageMakerExecutionRole
S3_BUCKET=your-ml-artifacts-bucket

Pipeline Configuration

Modify configs/{env}/pipeline_config.json:

{
  "pipeline_name": "YourMLPipeline",
  "model_package_group_name": "YourModelGroup",
  "processing_instance_type": "ml.m5.xlarge",
  "training_instance_type": "ml.m5.xlarge",
  "endpoint_instance_type": "ml.m5.large"
}

📊 Pipeline Steps

1. Data Preprocessing

• Input: Raw data from S3
• Processing: Feature engineering, data cleaning, train/val/test split
• Output: Processed datasets ready for training

2. Model Training

• Input: Processed training and validation data
• Training: Distributed training with hyperparameter optimization
• Output: Trained model artifacts

3. Model Evaluation

• Input: Trained model and test data
• Evaluation: Performance metrics calculation
• Output: Evaluation report and model quality assessment

4. Conditional Model Registration

• Condition: Model meets quality thresholds
• Registration: Automatic registration in SageMaker Model Registry
• Approval: Configurable approval workflow

5. Model Deployment

• Endpoint Creation: Automatic endpoint deployment for approved models
• Scaling: Auto-scaling configuration
• Monitoring: Data capture and model monitoring setup

🎛️ Model Management

Deploy a Model to Endpoint

from src.inference.deploy_endpoint import SageMakerEndpointManager

manager = SageMakerEndpointManager()

# Deploy model
predictor, endpoint_name = manager.deploy_model(
    model_data_url="s3://bucket/path/to/model.tar.gz",
    instance_type="ml.m5.large"
)

# Test endpoint
result = manager.test_endpoint(endpoint_name)

Model Registry Operations

# List model packages
aws sagemaker list-model-packages \
    --model-package-group-name TrafileaMLModelGroup-dev

# Approve a model
aws sagemaker update-model-package \
    --model-package-arn arn:aws:sagemaker:... \
    --model-approval-status Approved

📈 Monitoring and Observability

Built-in Monitoring

• CloudWatch Metrics: Training job metrics, endpoint metrics
• SageMaker Model Monitor: Data drift detection
• Pipeline Execution Tracking: Step-by-step execution monitoring

Custom Monitoring

# Enable model monitoring
from sagemaker.model_monitor import DefaultModelMonitor

monitor = DefaultModelMonitor(
    role=sagemaker_role,
    instance_count=1,
    instance_type='ml.m5.xlarge'
)

# Attach to endpoint
monitor.suggest_baseline(
    baseline_dataset=baseline_data_uri,
    dataset_format=DatasetFormat.csv(header=True)
)

🧪 Testing

# Run all tests
make test

# Run specific test
pytest tests/test_pipeline.py -v

# Test with coverage
pytest --cov=src tests/

🚀 Deployment

Development

make deploy-infra ENV=dev
make run-pipeline ENV=dev

Production

make deploy-infra ENV=prod
make run-pipeline ENV=prod

🔄 CI/CD Integration

GitHub Actions Example

name: SageMaker Pipeline CI/CD

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup AWS
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      
      - name: Deploy Infrastructure
        run: make deploy-infra ENV=prod
      
      - name: Run Pipeline
        run: make run-pipeline ENV=prod

🤝 Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Commit changes: git commit -m 'Add amazing feature'
4. Push to branch: git push origin feature/amazing-feature
5. Open a Pull Request

📝 License

Copyright © 2024 Trafilea

🆘 Support

• 📖 SageMaker Documentation
• 🎯 SageMaker Pipelines Guide
• 📧 Contact: mlops@trafilea.com

---

🎉 Ready to build production ML pipelines with SageMaker!