feat(load-balancer): implement Live Load Balancer runtime Docker infrastructure #45
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…astructure Add complete Docker infrastructure and testing support for LiveLoadBalancerSLSResource remote code execution: - **FastAPI Handler** (src/lb_handler.py): HTTP-based /execute endpoint for synchronous function execution - **Dockerfile-lb**: Docker image based on PyTorch CUDA runtime with FastAPI/uvicorn support - **Testing**: Comprehensive test suite with test-lb-handler.sh script mirroring test-handler.sh pattern - **CI/CD Pipeline**: Automated GitHub Actions jobs for building and pushing Load Balancer images (test, main, prod stages) - **Smoketest Support**: New make targets (smoketest-macos-lb-build, smoketest-macos-lb) for local Docker validation - **Documentation**: Complete guides on Load Balancer runtime and CI/CD pipeline architecture Key Features: - Direct HTTP POST to /execute endpoint with FunctionRequest JSON payloads - Support for both direct and RunPod-wrapped request formats - Full dependency installation (Python and system packages) - Port 8000 exposure for HTTP communication - All 14 handler tests passing locally; 13/14 passing in Docker container This enables local development and testing of Load Balancer-based serverless functions before production deployment.
Implement CPU-only version of Load Balancer runtime (Dockerfile-lb-cpu), mirroring the pattern of Dockerfile vs Dockerfile-cpu for queue-based workers. Changes: - Create Dockerfile-lb-cpu with python:3.12-slim base (no GPU/CUDA dependencies) - Add build-lb-cpu and smoketest-macos-lb-cpu make targets - Add docker-test-lb-cpu, docker-main-lb-cpu, docker-prod-lb-cpu CI/CD jobs - Update Docker_Build_Pipeline.md documentation to include CPU Load Balancer images - Support builds for development (:main tag) and releases (semantic versioning) This provides lightweight CPU-only option alongside GPU Load Balancer for deployments without GPU requirements.
…ndler The uvicorn CMD in both Dockerfile-lb and Dockerfile-lb-cpu was referencing 'handler:app' but the actual module is 'lb_handler.py'. This caused ASGI app loading errors when workers were deployed. Change both Dockerfiles to use 'lb_handler:app' so uvicorn can properly find and load the FastAPI app.
Remove unnecessary /health endpoint and update docstring. Keep /ping (required by RunPod) and /execute (for local dev). All 14 handler tests passing.
Update both Dockerfile-lb and Dockerfile-lb-cpu to expose and listen on port 80 instead of 8000, per RunPod Load Balancer specification. Changes: - EXPOSE 80 instead of 8000 - CMD uses --port 80 instead of --port 8000 - Update inline comments to reflect correct port
Update the if __name__ == '__main__' uvicorn server to use port 80, consistent with Dockerfile and RunPod Load Balancer specification.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR implements the complete Docker infrastructure and testing support for Load Balancer-based serverless function execution with HTTP endpoints. It enables local development and testing of functions via a /execute endpoint before production deployment.
Key changes:
- FastAPI-based HTTP handler with
/pinghealth check and/executeendpoints for remote function execution - New Dockerfiles (
Dockerfile-lb,Dockerfile-lb-cpu) supporting both GPU and CPU Load Balancer runtimes - Comprehensive CI/CD pipeline with automated testing, building, and pushing of Load Balancer images across test/main/production stages
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tetra-rp | Updated submodule reference to branch with Load Balancer resource implementation |
| src/lb_handler.py | New FastAPI handler providing /ping and /execute endpoints for HTTP-based function execution |
| src/test-lb-handler.sh | Test script for validating Load Balancer handler with comprehensive endpoint testing |
| src/test-handler.sh | Updated to use uv-managed Python for consistency |
| src/tests/test_lb_simple_function.json | Test fixture for simple addition function execution |
| pyproject.toml | Added FastAPI and uvicorn dependencies for HTTP handler support |
| Dockerfile-lb | GPU-enabled Load Balancer Docker image with PyTorch CUDA runtime |
| Dockerfile-lb-cpu | CPU-only Load Balancer Docker image |
| Makefile | New build/test targets for Load Balancer images and smoketests |
| .github/workflows/ci.yml | CI/CD jobs for testing and deploying Load Balancer images |
| docs/Load_Balancer_Docker_Infrastructure.md | Architecture documentation for Load Balancer runtime |
| docs/Docker_Build_Pipeline.md | CI/CD pipeline documentation |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…script and documentation This fixes the port inconsistency identified in PR #45 review comments. The Load Balancer uses port 80 by default (RunPod standard), but the test script and documentation incorrectly referenced port 8000. Changes: - Updated test-lb-handler.sh to use PORT=80 - Updated all documentation examples and references to use port 80 instead of 8000 - Aligns test infrastructure with actual Dockerfile configuration Resolves all 4 review comments on PR #45.
The --python flag specifies which Python interpreter to install packages to. Without it, uv pip install --system was returning success but not actually installing packages to the system site-packages. This caused import errors when trying to use installed dependencies. Fixes test_dependencies.json failures in CI/CD.
The --python flag specifies which Python interpreter to install packages to. Without it, uv pip install --system was returning success but not actually installing packages to the system site-packages. This caused import errors when trying to use installed dependencies. Fixes test_dependencies.json failures in CI/CD.
…/runpod-workers/worker-tetra into deanq/ae-1102-live-load-balancer
…tial package detection Implement differential installation to avoid unnecessary package reinstalls. The _filter_installed_packages method checks if packages are already installed before attempting installation, preventing redundant operations like reinstalling PyTorch in pytorch-based Docker images. Changes: - Filter out already-installed packages before installation - Simplify Docker installation to use standard uv pip and pip commands - Add package import check to detect existing installations - Handle package name variations (e.g., package-name vs package_name) Fixes test_dependencies.json failures in CI/CD.
…/runpod-workers/worker-tetra into deanq/ae-1102-live-load-balancer
The differential installation filter was interfering with package installation in Docker. Reverting to main's proven working approach which uses: - uv pip install --system (with accelerate_downloads) - pip install (without acceleration) This ensures numpy and other dependencies install correctly in the Docker runtime.
The previous version used 'uv run python3 handler.py' which created an isolated environment that didn't have access to system-installed packages. In Docker, packages are installed via 'uv pip install --system' to the system site-packages, so using 'uv run' prevented the tests from finding dependencies like numpy. This fix detects the environment and: - Uses 'python handler.py' in Docker (where python is available with system packages) - Uses 'uv run python3 handler.py' locally (where uv manages dependencies) Fixes: Test failures in docker-test CI/CD job
jhcipar
approved these changes
Jan 8, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implements complete Docker infrastructure and testing support for LiveLoadBalancerSLSResource remote code execution capabilities. This enables local development and testing of Load Balancer-based serverless functions with HTTP
/executeendpoint before production deployment.Key Components
src/lb_handler.py): HTTP-based/executeendpoint for synchronous function execution with/healthchecksrc/test-lb-handler.sh): Comprehensive test suite mirroringtest-handler.shpatternsmoketest-macos-lb-build,smoketest-macos-lb) for local Docker validationTechnical Details
/executeendpoint withFunctionRequestJSON payloadsTest Results
Test Plan
make test-lb-handlermake smoketest-macos-lbmake quality-checkRelated Issues
Implements infrastructure for runpod/tetra-rp#131