Provider Segmentation

About

A complete data clustering internship project using K-Means, Hierarchical Clustering, DBSCAN, Spectral Clustering, and Gaussian Mixture Model (GMM) Clustering to segment service providers into different groups based on selected features and also comparing each model's best possible performance.

---

Key Principles

Data clustering is an unsupervised machine learning technique that organizes and classifies different objects, data points, or observations into groups or clusters based on similarities or patterns. Unlike supervised learning, clustering does not rely on labeled data and instead aims to find natural groupings within the data.

Clustering is used to identify underlying trends, patterns, and outliers in a dataset. It can be applied in various scenarios, such as exploratory data analysis, preprocessing, and anomaly detection. Clustering helps in reducing the complexity of large datasets by grouping similar data points together, which can simplify further analysis and visualization.

---

Table of Contents

1st Phase - Exploratory Data Analysis (Data Cleaing & Transformation + Feature Engineering)

• Step 1 | Setup and Initialization
  • Step 1.1 | Importing Necessary Libraries
  • Step 1.2 | Loading the Dataset
  • Step 1.3 | Dataset Description
• Step 2 | Exploratory Data Analysis
  • Step 2.1 | Dataset Overview
  • Step 2.2 | Summary Statistics
  • Step 2.3 | Data Visualization
    • Univariate Analysis (Single Variable)
    • Bivariate Analysis (Two Variable)
    • Time-based Analysis
• Step 3 | Data Cleaning & Transformation
  • Step 3.1 | Handling Missing Values
  • Step 3.2 | Handling Duplicates
  • Step 3.3 | Treating Cancelled Bookings
  • Step 3.4 | Correcting Date Anomalies
  • Step 3.5 | Ensuring Data Consistency
    • Same-day Inconsistency
    • Status Inconsistency
    • Geographic Inconsistency
    • IDs Inconsistency
  • Step 3.6 | Outlier Treatment
• Step 4 | Feature Engineering
  • Step 4.1 | RFM Features
    • Step 4.1.1 | Recency (R)
    • Step 4.1.2 | Frequency (F)
    • Step 4.1.3 | Monetary (M)
  • Step 4.2 | Service Type Diversity
  • Step 4.3 | Behavioral Features
  • Step 4.4 | Cancellation Insights
  • Step 4.5 | Seasonality & Trends
  • Step 4.6 | Advanced Provider Features
  • Step 4.7 | Export Feature Engineer Data
    • Provider Dataset Description
• Step 5 | Data Preparation for Clustering Models

Clustering Pipeline

2nd Phase - Data Modeling & Analysis

• Step 1 | Config
• Step 2 | Setup
• Step 3 | Data Load (mock with numeric + categorical, or external)
  • Step 3.1 | Dataset Description
• Step 4 | Feature Mapping (with optional one-hot encoding)
• Step 5 | EDA
  • Step 5.1 | Feature Visualization
• Step 6 | Missing Values & Casting
  • Imputation Methods Theory
• Step 7 | Feature Engineering
• Step 8 | Feature Selection
• Step 9 | Outlier Handling
  • Outlier Handling Theory
• Step 10 | Scaling
  • Step 10.1 | Compare Clustering Across Two Scalers
• Step 11 | PCA
  • Principal Component Analysis Theory
  • Step 11.1 | 3D PCA Visualization
• Step 12 | Clustering Models
• Step 13 | Hyperparameter Sweeps
• Step 14 | Validation Metrics
  • Validation Metrics Theory
• Step 15 | Visualization (scatter + centroids)
• Step 16 | Cluster Profiling
  • Step 16.1 | Cluster Characteristic visualization
  • Step 16.2 | Cluster Naming (business interpretation)
• Step 17 | Export artifacts
  • Export Artifacts Theory

---

Main Process

• Exploratory Data Analysis (EDA)
• Data Preprocessing (Data Cleaning & Transformation)
• Feature Engineering (Feature Extraction & Selection)
• Data Preparation
• K-Means Clustering
• Hierarchical Clustering
• DBSCAN
• Spectral Clustering
• Gaussian Mixture Model Clustering
• Silhouette Score
• Davies-Bouldin Index
• Calinski-Harabasz Index

---

Getting Started

This project contains two Jupyter Notebooks that document the process and results of the internship work.
Since the dataset may be confidential, it is not included in this repository.

• provider_segmentation_eda.ipynb – Exploratory Data Analysis (EDA) of provider-related data.
• provider_segmentation_clustering.ipynb – Clustering process and resulting segmentation.

You can open these notebooks in Jupyter Notebook, JupyterLab, or Google Colab to review the workflow and outputs.

---

License

This project is licensed under the Apache-2.0 License - see the LICENSE file for details.