feat: Create E2E tests framework using PyTest (#677)

[1234-ad]

Description

This PR implements a comprehensive PyTest-based E2E testing framework for OPAL, addressing issue #677.

Changes

New Files Created

1. tests/e2e/conftest.py - PyTest configuration and fixtures

   • opal_keys: Generates authentication keys and tokens
   • policy_repo: Sets up Gitea and policy repository
   • opal_environment: Manages OPAL server/client containers
   • compose_command: Helper for docker-compose operations

2. tests/e2e/test_opal_e2e.py - Comprehensive test suite

   • TestOPALHealth: Health endpoint tests
   • TestOPALConnectivity: Server-client connectivity tests
   • TestOPALPolicyUpdates: Policy update tests (single & multiple)
   • TestOPALDataUpdates: Data update tests (single & multiple)
   • TestOPALStatistics: Statistics endpoint verification
   • TestOPALResilience: Broadcast channel recovery tests

3. tests/e2e/README.md - Comprehensive documentation

   • Framework overview and architecture
   • Usage instructions and examples
   • Test coverage details
   • Troubleshooting guide

4. tests/e2e/MIGRATION_GUIDE.md - Migration guide from bash

   • Feature mapping (bash → PyTest)
   • Best practices
   • Debugging tips

5. tests/e2e/requirements.txt - Python dependencies

6. .github/workflows/e2e-pytest.yml - CI/CD workflow

7. tests/__init__.py and tests/e2e/__init__.py - Package initialization

Features

✅ Complete Test Coverage - All functionality from app-tests/run.sh is covered:

• Health checks for server and client
• Client-server connectivity verification
• Policy bundle reception
• PubSub connection
• Static data loading
• Policy updates via git push
• Data updates via API
• Statistics endpoint
• Broadcast channel resilience
• No critical errors validation

✅ Better Maintainability

• Python-based tests are easier to read and maintain than bash
• Modular design with reusable fixtures
• Type hints for better IDE support
• Clear docstrings and comments

✅ Enhanced Debugging

• Detailed PyTest assertion messages
• Stack traces for failures
• Can run individual tests
• Verbose output options

✅ Flexible Execution

# Run all tests
pytest tests/e2e/

# Run specific test class
pytest tests/e2e/test_opal_e2e.py::TestOPALHealth

# Run with retries
pytest tests/e2e/ --reruns 3

# Run in parallel
pytest tests/e2e/ -n auto

✅ CI/CD Ready

• GitHub Actions workflow included
• Automatic cleanup on failure
• Test result artifacts

Testing

The framework has been designed to match all functionality from the existing bash script:

Test Category	Bash Script	PyTest Framework
Health Checks	✅	✅
Connectivity	✅	✅
Policy Updates	✅	✅
Data Updates	✅	✅
Statistics	✅	✅
Resilience	✅	✅

How to Run

# Install dependencies
pip install -r requirements.txt
pip install -r tests/e2e/requirements.txt

# Run tests
pytest tests/e2e/ -v -s

Benefits Over Bash Script

1. Modularity: Reusable fixtures and helper functions
2. Flexibility: Run individual tests or test classes
3. Better Error Messages: Detailed assertion failures
4. IDE Support: Autocomplete, refactoring, debugging
5. Parallel Execution: Run tests faster with pytest-xdist
6. Industry Standard: PyTest is widely used and well-documented
7. Extensibility: Easy to add new tests
8. Type Safety: Type hints for better code quality
9. Rich Ecosystem: Many plugins available
10. Better Debugging: pdb integration, verbose output

Migration Path

The bash script (app-tests/run.sh) can remain for backward compatibility while teams migrate to PyTest. The migration guide provides detailed mapping between bash functions and PyTest equivalents.

Documentation

• README.md: Complete framework documentation
• MIGRATION_GUIDE.md: Detailed migration guide from bash
• Inline docstrings: All fixtures and tests are documented

Closes

Closes #677

Checklist

• Created comprehensive PyTest E2E framework
• All bash script functionality covered
• Session-scoped fixtures for efficient setup/teardown
• Comprehensive test coverage (health, connectivity, policy, data, statistics, resilience)
• Detailed documentation (README, migration guide)
• GitHub Actions workflow
• Requirements file
• Type hints and docstrings
• Clear assertion messages
• Automatic cleanup

Additional Notes

This implementation provides a solid foundation for E2E testing that can be easily extended with:

• Performance benchmarking
• Security testing
• Multi-tenant testing
• Cedar policy engine tests
• Load testing
• Chaos engineering

The framework is production-ready and can be integrated into CI/CD pipelines immediately.

[netlify]

✅ Deploy Preview for opal-docs canceled.

Name	Link
🔨 Latest commit	955fd50
🔍 Latest deploy log	https://app.netlify.com/projects/opal-docs/deploys/697f0a0e2943a80008bc2cee