Add containerized testing framework for profile validation with TAP reporting, parallel execution and CI support

[Copilot]

• Create Containerfile for Fedora-based test environment
• Create scripts/tests.sh with container management utilities and assert function
• Add tests.sh to _template profile with three test cases
• Add tests.sh to profile 0 with three test cases
• Add .gitignore to exclude test reports
• Test the implementation manually with podman
• Verify all test cases work correctly
• Add comprehensive testing documentation to README.md
• Fix documentation to match actual script behavior
• Address code review feedback:
  • Install prerequisites in Containerfile using dnf
  • Run check function from get.sh in Containerfile
  • Use git to determine repo root instead of dirname/..
  • Remove fallback to base image if build fails
  • Move common test cases to global test runner
  • Add profile-specific tests for profile 0 (brew and zsh)
  • Remove redundant example from README
• Add support for both podman and docker (prefers podman, falls back to docker)
• Create GitHub Actions workflow "test" that runs tests on push/PR and uploads test report as artifact
• Refactor to run each test case in its own isolated container
• Add parallel test execution where possible (syntax checks, independent install/uninstall tests)
• Implement file-based locking for thread-safe report writing
• Exit with status code equal to number of failed tests (count [FAIL] in report)
• Move failure handling into assert function - tests continue automatically without || true in profile code
• Profile-specific tests now share a single container with install script pre-run for realistic testing
• Deduplicate run_test_case functions - container_id is now an optional last parameter
• Add SETUP step to test report showing install script execution logs for profile-specific tests
• Fix profile 0 uninstall script to exit with code 0 when brew is not installed
• Fix profile 0 brew test to run in zsh context where PATH is configured
• Refactor tests.sh for improved clarity and maintainability:
  • Extract helpers (exec_in_container, write_test_result)
  • Consolidate initialization at top
  • Simplify common test logic with && chaining
  • Separate concerns (run_profile_specific_tests)
  • Reduce line count from 301 to 245 while maintaining all functionality
• Migrate to TAP (Test Anything Protocol) format:
  • Change report format from custom text to standard TAP
  • Output format: ok N - description / not ok N - description
  • Add TAP version header and test plan (1..N)
  • Include diagnostic info with # prefix
  • Update report file from .txt to .tap
  • Update CI workflow to use TAP report
  • Benefits: standard format, tool support, better CI integration
• Clean up test infrastructure:
  • Remove obsolete test_report.txt from .gitignore
  • Remove section heading comments from scripts/tests.sh
  • Use plain language in output messages (no decorative separators)
  • Upload test report to GitHub Actions job summary instead of artifacts
• Add POSIX-compliant .profile for profile 0:
  • Translate .zprofile logic to POSIX sh syntax
  • Use case statements instead of bash [[ ]] conditionals
  • Ensures PATH and brew environment are available in sh shells
  • Fixes test failures when running tests with sh instead of zsh

Original prompt

create a testing script for the profiles.

the script should use podman containers based on fedora.

we need a containerfile with the basic setup. here we install pre-requisites and mount the repository so that the tests can be executed.

all tests will run on this image.

each profile will have a tests.sh file inside of it with functions.

each test will run it's own container and cleanup after itself afterwards.

the logic to start a container and clean it up should be written in a global script 'sctipts/tests.sh'.

we should have a utility function called "assert" that receives a command and a description. it runs the command in the container, and if the exit code is zero it assumes it's a passed test. otherwise it assumes it's a failed test.

the assert command should place the output of the command and the final state in a central report file.

keep the report file in a human readable format that can be quickly scanned.

do not run this on the ci, set it up so it can be run on a developer machine with a single command.

optionally, add the ability to run the tests for a single profile.

you are testing the profile, not the entire repository. do not use global scripts. focus on building the test cases for each profile individually.

add a test that ensures the install script of the profile exits successfully when run once.

add a test that ensures the install script of the profile exits successfully when run twice in a row.

add a test that ensures the install script of the profile exits successfully when run once and then the uninstall script exits successfully.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Stale review