Task Overview

Develop a Node.js application that extracts data from various APIs (e.g., gold prices, goods prices, crypto P2P prices) and stores the data in a PostgreSQL database. The application should run as a cron job that automatically fetches data every 24 hours, and it should include an endpoint to retrieve data for a specified date. The system should be designed with the flexibility to easily add new APIs or data sources.

Objective

• Build a scalable and flexible data extraction service using Node.js.
• Integrate with multiple external APIs to collect price data daily and store it in a PostgreSQL database.
• Implement a RESTful endpoint to fetch stored data for a given date.
• Design a robust database schema to support various data types and retrieval requirements targeting future additions of new API sources.

Technical Requirements

Language & Framework

• Node.js
• Express
• TypeScript

Database

• PostgreSQL with an ORM like Knex, Sequelize, or Drizzle

APIs to Integrate

• Gold prices: https://ceyloncash.com/api/goldrates/
• P2P rates: https://nisal.me/p2p/post.php?asset=USDT&fiat=LKR&tradeType=BUY
• Consumer prices:
  • https://api.welandapola.com/api/prices?pagination[pageSize]=1000&filters[date][$between]=2024-09-04&filters[date][$between]=2024-09-05&filters[market][$eq]=narahenpita&filters[type][$eq]=retail
  • https://prices.welandapola.com/

System Architecture

• The application should run as a cron job, scheduled to execute every 24 hours.
• Integrate with multiple APIs to fetch data, handle responses, and log results.
• Store data in a PostgreSQL database with a schema designed to support the various data from each API.

Service Flexibility

• Develop a modular design to easily integrate new services/APIs.

Database Design

• Define a database architecture that supports storing data from multiple APIs.
• Ensure the schema is structured to efficiently store, index, and retrieve data based on date or other relevant filters.
• The schema should be designed to handle different data structures from each API while maintaining performance.
• Write necessary migration files.

Endpoint for Data Retrieval

• Implement a RESTful endpoint that allows users to fetch data gathered for a specified date.
• Ensure the endpoint is optimized for querying historical data from the database.

Error Handling

• Implement error handling for API calls, connection issues, and data validation errors.
• Log errors and exceptions to track the daemon’s performance and any issues.

Logging & Monitoring

• Set up detailed logging for each API call, including successes and failures.

Performance Considerations

• Implement retry mechanisms for failed API requests with exponential backoff.

Deliverables

• Complete Node.js application with integrated APIs.
• Defined database schema and migration scripts for PostgreSQL.
• RESTful endpoint for fetching data for a specified date.
• Error handling and logging implementations.
• Tests to validate API integrations, data storage, and endpoint functionality.

Evaluation Criteria

• Code quality, modularity, and maintainability.
• Database design and ability to efficiently store and query data.
• Effectiveness and reliability of data extraction and storage.
• Endpoint performance and ability to handle various query parameters.
• Quality of error handling, logging, and documentation.