Skip to content
#

real-time-data-pipeline

Star

Here are 6 public repositories matching this topic...

Language: All
Filter by language
All 6 Python 4 Java 1 Jupyter Notebook 1

kishlayjeet / Stock-Market-Real-Time-Data-Pipeline-with-Apache-Kafka-and-Cassandra

Star 22

A end-to-end real-time stock market data pipeline with Python, AWS EC2, Apache Kafka, and Cassandra Data is processed on AWS EC2 with Apache Kafka and stored in a local Cassandra database.

python aws real-time kafka cassandra ec2 pipeline etl data-engineering stock-market aws-ec2 ec2-instance apache-kafka kafka-streams kafka-streaming data-pipeline etl-pipeline data-engineering-pipeline stock-market-data-pipeline real-time-data-pipeline
  • Updated Jun 7, 2023
  • Python

aymane-maghouti / Real-Time-Data-Pipeline-Using-Kafka

Star 5

This project implements a real-time data pipeline using Apache Kafka, Python's psutil library for metric collection, and SQL Server for data storage. The pipeline collects metrics data from the local computer, processes it through Kafka brokers, and loads it into a SQL Server database. Additionally, a real-time dashboard is created using Power BI.

python real-time data-visualization data-collection apache-kafka powerbi data-streaming real-time-data-pipeline
  • Updated Oct 31, 2023
  • Python

aymane-maghouti / Real-Time-Streaming-Kafka-Debezium-Spark-Streaming

Star 3

This project demonstrates real-time data streaming and processing architecture using Kafka, Spark Streaming, and Debezium for capturing CDC (Change Data Capture) events. The pipeline collects transaction data, processes it in real time, and updates a dashboard to display real-time analytics for smartphone data.

notifications python java docker web-development kafka spring-boot dashboard reactjs spark-streaming data-analytics mysql-database postgresql-database data-processing change-data-capture debezium real-time-systems real-time-data-pipeline
  • Updated Oct 24, 2024
  • Java

quixio / community-highlights

Star 1

Collecting highlights from the Quix community and social media in the form of interesting questions, comments, challenges, solutions and insights

python kafka etl stream-processing data-engineering kafka-consumer kafka-producer python-kafka data-pipeline streaming-etl stream-processor realtime-data event-driven-architecture kafka-python kafka-producer-consumer streaming-pipeline data-intensive-applications realtime-etl real-time-data-pipeline real-time-data-processing
  • Updated Aug 30, 2024
  • Python

stereosky / community-highlights

Star 0

Collecting highlights from the Quix community and social media in the form of interesting questions, comments, challenges, solutions and insights

python kafka etl stream-processing data-engineering kafka-consumer kafka-producer python-kafka data-pipeline streaming-etl stream-processor realtime-data event-driven-architecture kafka-python kafka-producer-consumer streaming-pipeline data-intensive-applications realtime-etl real-time-data-pipeline real-time-data-processing
  • Updated Jul 24, 2024
  • Python

Improve this page

Add a description, image, and links to the real-time-data-pipeline topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the real-time-data-pipeline topic, visit your repo's landing page and select "manage topics."

Learn more