Skip to content

Comments

Handle large diffs without stdout maxBuffer overflow#57

Merged
LouisLetcher merged 9 commits intomainfrom
feature/total_refactor
Sep 15, 2025
Merged

Handle large diffs without stdout maxBuffer overflow#57
LouisLetcher merged 9 commits intomainfrom
feature/total_refactor

Conversation

@LouisLetcher
Copy link
Collaborator

@LouisLetcher LouisLetcher commented Sep 12, 2025

Pull Request

Description

Replace usage of child_process.exec with child_process.spawn for all Git commands. Streaming stdout/stderr avoids “stdout maxBuffer length exceeded” on large diffs and makes state retrieval resilient and memory‑efficient. Also consolidates process/error handling and adds optional timeouts/cancellation to improve reliability.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Performance improvement
  • Code refactoring

Changes Made

  • Switched Git invocations from exec to spawn, streaming output to prevent buffer overflows.
  • Implemented unified process runner with:
  • incremental stdout/stderr handling
  • exit code propagation and structured errors
  • optional timeout and AbortController-based cancellation
  • Updated all call sites to consume streamed output (line-by-line parsing where applicable).
  • Improved logging of long-running operations without buffering large data in memory.
  • Added safeguards for large output (e.g., big diffs, blame, logs).
  • Updated docs and JSDoc/comments to reflect streaming behavior and API shape.
  • Adjusted tests to support streaming and large-output scenarios; added regression cases for “maxBuffer exceeded”.

Testing

  • Tests pass locally
  • New tests added (if applicable)
  • Manual testing completed

Checklist

  • Code follows project style guidelines
  • Self-review completed
  • Documentation updated (if needed)
  • No breaking changes (or documented)

Related Issues

Fixes #(issue number)

Use child_process.spawn to run git commands and stream output instead of
buffering with exec. Prevents "stdout maxBuffer length exceeded" on large
diffs and makes state retrieval reliable.
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex Review: Here are some suggestions.

Reply with @codex fix comments to fix any unresolved comments.

About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".

LouisLetcher and others added 8 commits September 12, 2025 17:54
…in permissions

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Signed-off-by: Manuel H. <36189959+LouisLetcher@users.noreply.github.com>
…in permissions

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Signed-off-by: Manuel H. <36189959+LouisLetcher@users.noreply.github.com>
…in permissions

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Signed-off-by: Manuel H. <36189959+LouisLetcher@users.noreply.github.com>
- Data: Add only_cached flag to Polygon/Tiingo fetch; return cached data when present, raise on cache miss to support offline runs; propagate through main/registry.
- CI: Add Markdownlint step; provide GITHUB_TOKEN to Gitleaks; remove custom CodeQL workflow and rely on GitHub’s Default setup (README updated).
- Pre-commit: Set COVERAGE_FILE and PYTHONPATH in pytest hook to avoid polluting repo and fix imports in hooks/CI.
- Pass --exit-code 0 to gitleaks so detections don’t fail the job while still generating SARIF
- Upload SARIF only if present (hashFiles check) to avoid unnecessary step failures
- Add tests/conftest.py to prepend repo root to sys.path for reliable imports in tests (e.g., src.*)
- Add HealthReporter and export a health report after runs (wrapped in try/except) to surface failures without impacting execution
- .env.example: add HOST_STRATEGIES_PATH and API keys for Finnhub, Twelve Data, Alpha Vantage
- README: document symbol mapping across providers, expand provider coverage (incl. futures), and clarify report output paths
- Lint: update Ruff rules (stop ignoring UP038; ignore B008 instead)

Why: improve run diagnostics and observability, ease local strategy development, and prepare for broader data source support while tightening linting/typing standards.
Replace tuple of ccxt exceptions with union (|) syntax in error
handling to modernize code for Python 3.10+, improve readability,
and align with linters (e.g., pyupgrade/Ruff). No functional changes.
@LouisLetcher LouisLetcher merged commit 9250c44 into main Sep 15, 2025
8 checks passed
@LouisLetcher LouisLetcher deleted the feature/total_refactor branch September 15, 2025 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants