Add comprehensive CODEX-REVIEW.md repository audit

[adrianwedd]

Motivation

• Create a single, reviewable audit that documents the repository's purpose, architecture, code health, documentation gaps, technical debt, and modernization opportunities.
• Surface critical risks and prioritized remediation steps so maintainers can triage security, packaging, and operational issues quickly.
• Provide concrete pointers (file paths and line references) and questions for maintainers to clarify ambiguous design and config choices.

Description

• Add CODEX-REVIEW.md containing an Executive Summary, Critical Issues, Priority Improvements (quick/medium/substantial), Latent Risks, Questions for the maintainer, and a "What's Actually Good" section.
• Key findings highlighted include API auth defaults exposing a test-key, a missing pydantic_settings runtime dependency, CLI vs docs/packaging mismatches, and undocumented environment variables such as NETWORK_* and SENTRY_DSN.
• The document points out duplicated/parallel scrapers (scripts/scrape_repos.py vs agentic_index_cli/internal/scrape.py) and inconsistent coverage/packaging thresholds across pyproject.toml, setup.cfg, and scripts/coverage_gate.py.
• The review references concrete files and line ranges (e.g., agentic_index_api/config.py, agentic_index_api/server.py, agentic_index_cli/github_client.py, requirements.txt, pyproject.toml) to aid fast follow-up.

Testing

• This is a documentation-only change and does not modify runtime code paths, so no automated tests were executed as part of this change.
• The change is limited to adding CODEX-REVIEW.md and does not affect existing tests or CI configuration.

---

Codex Task