Skip to content

Conversation

@bplatz
Copy link
Contributor

@bplatz bplatz commented Oct 10, 2025

Summary

Adds HTTP streaming query support using NDJSON (Newline-Delimited JSON) format for efficient handling of large result sets. This enables progressive result rendering and reduces memory pressure on both server and client.

Implementation

Core Changes

  • handlers/ledger.clj: Implemented ndjson-streaming-body function that converts core.async channels to Ring StreamableResponseBody, with proper exception handling and metadata support
  • Handler routing: Modified query handler to detect Accept: application/x-ndjson header and route to streaming vs. buffered execution
  • HTTP 206 status: Returns 206 Partial Content for streaming responses to bypass response coercion middleware

Streaming Behavior

  • Results emitted as tuples (arrays) for efficiency: ["Alice", 30]
  • Each result written as NDJSON line (one JSON value per line)
  • Optional metadata tracking: final {"_fluree-meta":{"status":200,"fuel":...,"time":...}} line when tracking headers are present
  • Stream closes when all results are emitted

Dependencies

  • consensus/events.clj: Updated imports from deprecated fluree.db.ledger to fluree.db.transact namespace
  • consensus/raft/handlers/new_commit.clj: Updated imports to use fluree.db.transact
  • deps.edn: Updated fluree/db dependency to SHA 74fcc49d1154146cc0e66126322f3ad5b0047969
  • task_test.clj: Replaced deprecated transact! with update! API

Testing

  • streaming_query_test.clj: Comprehensive integration tests covering:
    • Basic NDJSON streaming with tuple results
    • Metadata tracking (fuel, time, policy)
    • Query LIMIT support
    • Comparison with buffered query behavior
    • 21 assertions, all passing

Documentation

  • README.md: Added streaming query section with curl examples showing:
    • Basic NDJSON streaming usage
    • Query tracking headers for metadata
    • Key differences from buffered queries

Dependencies

This implementation builds on the fluree/db streaming query API introduced in:

Test Results

1 tests, 21 assertions, 0 failures

All existing tests continue to pass with no regressions.

- Introduced a new `ndjson-streaming-body` function to handle streaming responses in NDJSON format.
- Updated the query handler to support streaming queries based on the `Accept` header.
- Modified transaction handling to use the `transact` namespace instead of `ledger`.
- Added integration tests for streaming queries, including metadata tracking and limit functionality.
- Updated dependencies and fixed related test cases.
@bplatz bplatz requested a review from a team October 10, 2025 02:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants