Orchestration module

[Villarley]

Orchestration Module - Complete aggregation pipeline

📝 Description

Create an orchestration module that integrates all previous services (reception, normalization, outlier detection, aggregation, storage) into a cohesive pipeline. This module should coordinate the data flow from reception to final storage, handling errors and retries at each stage.

Context: Now that all individual components are implemented, we need an orchestrator to connect and coordinate them. This is the "brain" of the aggregator that decides what to do with each received price.

Tech Stack:

• NestJS Module with providers
• RxJS for reactive pipeline
• Event-driven architecture
• TypeScript with strict types

✅ Requirements

• Implement an AggregatorOrchestrationService that coordinates the complete pipeline
• Flow: Reception → Normalization → Outlier Detection → Aggregation → Storage
• Implement event-driven architecture (each stage emits events)
• Handle errors at each stage without stopping the complete pipeline
• Implement circuit breaker to prevent failure cascades
• Add detailed logging of the flow of each price
• Implement throughput and latency metrics per stage
• Configure aggregation intervals via environment variables
• Allow both real-time and batch processing

🎯 Acceptance Criteria

• The complete pipeline executes correctly from start to finish
• Each received price goes through all stages in order
• Errors at one stage are handled without crashing the pipeline
• Events are emitted at each stage for observability
• Circuit breaker is implemented that stops processing if there are many failures
• Latency and throughput metrics are logged per stage
• Integration tests of the complete pipeline exist (>75% coverage)
• The complete flow is documented with diagrams

📁 Expected files to change/structure

• apps/aggregator/src/modules/aggregator-orchestration.module.ts - Main module
• apps/aggregator/src/services/orchestration.service.ts - Orchestrator service
• apps/aggregator/src/services/orchestration.service.spec.ts - Tests
• apps/aggregator/src/events/ - Event definitions
  • price-received.event.ts
  • price-normalized.event.ts
  • price-aggregated.event.ts
• apps/aggregator/src/middleware/circuit-breaker.middleware.ts - Circuit breaker
• apps/aggregator/src/app.module.ts - Import orchestration module
• apps/aggregator/README.md - Document pipeline and architecture
• apps/aggregator/.env.example - Configuration variables

[drips-wave]

This issue has been added to the Stellar Wave Program and is worth 200 Points.

🧑‍💻 Interested in contributing? Apply to work on this issue on Drips Wave, earn points, and receive rewards.

Warning

Only applications submitted via the apply link above will be considered. Please do not apply by commenting on the issue or opening a PR directly.

🤠 Repo maintainers: You can manage this issue's Points and Complexity on your Maintainer Dashboard here. Please keep an eye on applications and respond quickly 💙

ℹ️ Learn more about Drips Wave

[Chucks1093]

@Chucks1093 has applied to work on this issue as part of the Stellar Wave Program's 1st wave.

Hello Villarley, I’m a full stack engineer and I’ve wired multi-stage pipelines like this in production before. Here’s my approach.

• Add apps/aggregator/src/modules/aggregator-orchestration.module.ts and wire it into apps/aggregator/src/app.module.ts, injecting the existing reception, normalization, outlier detection, aggregation, and storage providers so the orchestration layer only coordinates, not reimplements logic.
• Implement apps/aggregator/src/services/orchestration.service.ts as an RxJS pipeline that consumes a PriceReceivedEvent, runs each stage in strict order, emits stage events under apps/aggregator/src/events/, and supports both real-time and batch by switching the source stream and aggregation window.
• Handle errors per stage using catchError and a dead-letter style event payload so one bad price does not stop the stream, and include retry with backoff only where it makes sense (network or storage), not for deterministic validation failures.
• Add apps/aggregator/src/middleware/circuit-breaker.middleware.ts to track failure rates and short-circuit processing when thresholds are breached, then automatically half-open after a cooldown to prevent failure cascades.
• Instrument throughput and latency per stage with simple timers and counters, log with a requestId or priceId across all events, and add integration tests in apps/aggregator/src/services/orchestration.service.spec.ts that validate ordering, event emission, error isolation, circuit breaker behavior, and coverage above 75%, plus document the full flow in apps/aggregator/README.md and .env.example for intervals and modes.

This should be completed within about 18 to 26 hours.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Chucks1093 to this issue.

[emperorsixpacks]

@emperorsixpacks has applied to work on this issue as part of the Stellar Wave Program's 1st wave.

I'm Andrew, a fullstack developer with extensive experience in building robust and efficient systems. I would love to contribute to this issue and provide a high-quality solution. I'm available to start immediately.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @emperorsixpacks to this issue.

[soomtochukwu]

@soomtochukwu has applied to work on this issue as part of the Stellar Wave Program's 1st wave.

Yo @Villarley, please assign #21 to me. I'm hyped to build that slick Orchestration module in NestJS, wiring up the full pipeline (reception → normalization → outlier detection → aggregation → storage) with RxJS events, bulletproof error handling + retries, circuit breaker, per-stage logging/metrics, configurable intervals, real-time/batch support, integration tests over 75% coverage, and clean docs with diagrams. Let me lock this in and ship it smooth 🙏

ℹ️ Repo Maintainers: To accept this application, review their application or assign @soomtochukwu to this issue.

[JerryIdoko]

@JerryIdoko has applied to work on this issue as part of the Stellar Wave Program's 1st wave.

Hi, I would like to take on this issue.

I have reviewed the requirements and can implement the Aggregator Orchestration Service using NestJS and RxJS to ensure a robust, reactive pipeline.

Implementation Plan:

Reactive Flow: Use RxJS streams to manage the full lifecycle (Reception → Storage), handling backpressure and concurrency efficiently.

Resilience: Implement the Circuit Breaker pattern to isolate failures and ensure the pipeline is non-blocking.

Observability: Emit events at every stage (e.g., price.normalized) and integrate logging for latency/throughput metrics.

Quality: Deliver full integration tests (>75% coverage) and architectural documentation.

I am available to start immediately.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @JerryIdoko to this issue.

[Mosas2000]

@Mosas2000 has applied to work on this issue as part of the Stellar Wave Program's 1st wave.

I’ll like to take up this task, I’m a fullstack developer keen on impact and proven with a track record

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Mosas2000 to this issue.

[Amarjeet325]

@Amarjeet325 has applied to work on this issue as part of the Stellar Wave Program's 1st wave.

Hi maintainers 👋

I’d like to work on this issue.
I’m a full-stack developer and have previously contributed to multiple open-source projects.
Could you please assign this issue to me?

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Amarjeet325 to this issue.

[tali-creator]

@tali-creator has applied to work on this issue as part of the Stellar Wave Program's 1st wave.

I’ve carefully gone through the issue description and taken time to understand the existing codebase and expected outcome. Based on my experience and the scope of this task, I’m confident I fully understand what needs to be done and how it fits into the project’s overall architecture.

I’m a full-stack Stark developer with 4+ years of hands-on experience, and I’m very confident in my ability to deliver a solid, production-ready solution. I’m 100% certain I can complete this task within 24 hours while maintaining code quality, clarity, and alignment with the project’s standards. I’d be glad to take this on and contribute effectively if assigned.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @tali-creator to this issue.

[ryzen-xp]

@ryzen-xp has applied to work on this issue as part of the Stellar Wave Program's 1st wave.

Hi maintainers 👋
I’m a blockchain developer with strong expertise in Stellar, Rust,TS and Stellar SDK integrations. I’ve worked on multiple Stellar-based smart contract and backend systems.

I understand the scope of this issue and can deliver a reliable, production-quality solution with proper tests and documentation.
Plz assign me !!

ETA: 12 hours

ℹ️ Repo Maintainers: To accept this application, review their application or assign @ryzen-xp to this issue.

[ryzen-xp]

@ryzen-xp has applied to work on this issue as part of the Stellar Wave Program's 1st wave.

Hi maintainers 👋
I’m a blockchain developer with strong expertise in Stellar, Rust,TS and Stellar SDK integrations. I’ve worked on multiple Stellar-based smart contract and backend systems.

I understand the scope of this issue and can deliver a reliable, production-quality solution with proper tests and documentation.
Plz assign me !!

ETA: 12 hours

ℹ️ Repo Maintainers: To accept this application, review their application or assign @ryzen-xp to this issue.

[Georgechisom]

@Georgechisom has applied to work on this issue as part of the Stellar Wave Program's 1st wave.

Hi, I love to work on this issue and deliver an excellent result. Please help assign it to me.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Georgechisom to this issue.

[ShantelPeters]

@ShantelPeters has applied to work on this issue as part of the Stellar Wave Program's 1st wave.

I would like to work on this issue , kindly assign please , i'm ready to work .

ℹ️ Repo Maintainers: To accept this application, review their application or assign @ShantelPeters to this issue.

[caxtonacollins]

@caxtonacollins has applied to work on this issue as part of the Stellar Wave Program's 1st wave.

Hi 👋
I will love to work on this issue please assign

ℹ️ Repo Maintainers: To accept this application, review their application or assign @caxtonacollins to this issue.

[Lynndabel]

@Lynndabel has applied to work on this issue as part of the Stellar Wave Program's 1st wave.

can i work on this please? I have worked on something similiar.ETA: 10hours

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Lynndabel to this issue.