Sample codes/functions of pyspark

This repository contains an end-to-end real-time YouTube comments sentiment analysis solution. It uses Azure Event Hub for data ingestion, Azure Data Factory for orchestration, and Databricks for data processing with VADER for sentiment analysis. The pipeline outputs results to Delta Lake for scalable querying and storage.

This repository contains a data engineering project analyzing global earthquake events. Utilizing Microsoft Fabric, PySpark, and Power BI, it automates data fetching and cleaning from the USGS Earthquake Catalog and provides dynamic visualizations to uncover insights.

The IPL Data Analysis project aims to explore and analyze the Indian Premier League (IPL) data using PySpark for data processing and Matplotlib and Seaborn for data visualization. The goal is to derive actionable insights into player performances, match trends, and overall league dynamics.

This project presents a comprehensive data pipeline designed to predict customer churn using historical customer data. By leveraging Hadoop and PySpark, this pipeline efficiently processes large datasets, performs feature engineering, and trains a machine learning model to identify customers at risk of leaving.

APACHE SPARK: Data Analysis, Transformation, and Visualisation with PySpark, IPL Data Analysis

Azure projects - End to End Data Engineering Project with medallion architecture using Azure Data Factory & Azure Databricks. Azure Serverless/Logical DataWarehouse using Azure Synapse Analystics to demo CETAS, Data Modeling, Incremental loading, CDC and Sql Monitoring the data processing connected to Power BI

In this project I have performed complete analysis on a psl data of 2017 , I uploaded the data on the AWS s3 bucket , then load the data in databricks and then applied various transformations on the datasets to get various insights from the data.

This data project can be used as a take-home assignment to learn Pyspark and Data Engineering.

Implementation of K-means,Bisecting K-means and Decision Tree in PySpark on the Iris Dataset.

ORM for Apache Spark and DataFrames schema manager

ETL (Extract, Transform, Load) job using PySpark - submodule

Leverage the power of Apache Spark for large-scale data processing and analysis

CekatanBiz is Software Tools Data Analyst,Business Analyst,and Business Intelligence. Developed using Python.

Projet de création d'un datatlake sur le thème des jeux vidéos. Deux sources de données : API Kaggle (dataset de jeux avec dates de sorties et évaluation) + API Twitter(commentaires sur la base des hashtags des noms des jeux récupérés avec du code Python).

PySpark Job that runs in Dataproc cluster, loads data from Cloud Storage to BigQuery table.

Machine Learning using Pyspark