PR-Verify is an automated LLM agent for reviewing and verifying Github PRs.
We verify that contributor code works as intended, by generating unit tests and executing the code with LLM-generated "oracle verifiers".
Won "Best Pear Hack" at OpenAI x PearVC hackathon.
The workflow is runs on GH Actions:
- Event triggers on label "pr-verify"
- Runner generates unit tests using for the PR, checks if it is valid Python code using guardrails
- Runner tests the merged code on the unit tests in subprocess sandbox
- Creates a synopsis (with threat score and analysis) on the PR (todo)
the quick & easy setup:
- In repository settings
- Add OPENAI_AI_KEY to repo secrets
- Enable GH actions push access
- Copy over GH action:
.github/workflows/pr-verify.yml
That's it! Just add the pr-verify label and let PR-Verify do the rest.
- GH actions workflow, spin up docker container (code execution sandbox)
- automated repo installation setup (infer instructions from readme)
- synopsis/threat analysis comment on PR
- merging conflicts in feature branch?
- PR-Verify on GH Action marketplace