fix(ci): resolve disk space issues and optimize Docker image sizes

[deanq]

Summary

This PR resolves CI/CD disk space failures and optimizes Docker image sizes through intelligent build-essential management.

Changes

1. Fix CI/CD Disk Space Exhaustion

Problem: docker-test-lb job failing with "no space left on device" while building PyTorch CUDA images (https://github.com/runpod-workers/worker-tetra/actions/runs/20802819754)

Root Cause:

• Cleanup only ran on PRs (skipped main branch pushes)
• Manual cleanup provided only ~17-19GB free space
• PyTorch CUDA builds need ~25-30GB

Solution:

• Removed conditional cleanup that skipped main branch
• Upgraded all CUDA image builds to jlumbroso/free-disk-space action
• Added docker system prune -af to clear build cache
• Added df -h for disk space visibility

Result: 14GB → 34-39GB available (sufficient for all builds)

2. Auto-Install build-essential When Needed

Problem: If build-essential removed from images, packages needing compilation would fail with cryptic gcc errors

Solution: Intelligent detection + automatic retry

• Detect pip failures due to missing compilers (gcc, g++, cc not found)
• Automatically install build-essential via system_dependencies
• Retry pip install transparently
• User sees seamless success

Implementation:

• _needs_compilation(): Detects 14 error patterns (gcc not found, distutils errors, etc.)
• Auto-retry with nala-accelerated build-essential installation
• Comprehensive error handling and logging

Testing:

• 11 new unit tests covering detection and retry logic
• All 127 tests passing (77.30% coverage)
• Tests verify error pattern matching, auto-retry flow, nala acceleration
• Tests verify no false positives on unrelated errors

3. Remove build-essential from Base Images

Rationale:

• 99% of Python packages have pre-built manylinux wheels
• Users needing compilation get auto-install (30-60s one-time cost)
• Optimizes for common case while handling edge cases transparently

Changes: Removed build-essential from all 4 Dockerfiles

• Dockerfile (GPU)
• Dockerfile-cpu
• Dockerfile-lb (Load Balancer GPU)
• Dockerfile-lb-cpu

Preserved Dependencies:

• curl: Required for uv installation
• ca-certificates: Required for HTTPS
• nala: Parallel downloads for system packages
• git: Common for pip installs from repos

Measured Impact:

• CPU images: 844MB → 588MB (256MB / 30% reduction)
• GPU images: ~8.24GB → ~7.84GB (expected ~400MB / 5% reduction)

User Experience

For 99% of users (pre-built wheels):

• ✅ Smaller images, faster cold starts
• ✅ Zero compilation delay
• ✅ No changes needed

For 1% of users (packages needing compilation):

• ✅ Transparent auto-install of build-essential
• ✅ ~30-60s one-time installation (nala-accelerated)
• ✅ No manual system_dependencies needed
• ✅ Clear logging of what's happening

Examples

Before (Manual intervention required):

@remote(
    dependencies=["git+https://github.com/user/package.git"],
    system_dependencies=["build-essential"],  # User must know to add this
)

After (Automatic):

@remote(
    dependencies=["git+https://github.com/user/package.git"],
    # Auto-detects compilation need and installs build-essential
)

Breaking Changes

None - behavior is backward compatible and more user-friendly.

Testing

• ✅ All 127 unit tests passing
• ✅ 11 new tests for auto-retry feature
• ✅ 77.30% code coverage
• ✅ All linting and formatting checks pass
• ✅ CI/CD pipeline should now complete successfully

Related Issues

Fixes https://github.com/runpod-workers/worker-tetra/actions/runs/20802819754

[Copilot]

Pull request overview

This PR resolves CI/CD disk space failures during Docker image builds and optimizes image sizes by removing pre-installed build-essential from base images while implementing automatic detection and installation when compilation is needed.

Key Changes:

• Upgraded CI/CD disk cleanup strategy from manual cleanup to jlumbroso/free-disk-space action for CUDA image builds
• Implemented intelligent build-essential auto-installation with retry logic when packages require compilation
• Removed build-essential from all four Dockerfiles to reduce image sizes by ~400MB for CPU images and ~5% for GPU images

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
.github/workflows/ci.yml	Enhanced disk space management for CUDA builds using jlumbroso/free-disk-space action and added docker system prune
src/dependency_installer.py	Added _needs_compilation() method and auto-retry logic to detect and handle missing compiler errors
tests/unit/test_dependency_installer.py	Added 11 comprehensive unit tests for compilation detection and auto-retry functionality
Dockerfile	Removed build-essential from GPU base image with explanatory comments
Dockerfile-cpu	Removed build-essential from CPU base image with explanatory comments
Dockerfile-lb	Removed build-essential from load balancer GPU base image with explanatory comments
Dockerfile-lb-cpu	Removed build-essential from load balancer CPU base image with explanatory comments

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.