Deepfake Detection System (Transformer Edition)

A Twin-Stream Deepfake Detection System using Vision Transformers (ViT-B/16 / MobileNetV3) to analyze both Facial and Background inconsistencies.

Tech Stack

Features

• Twin-Stream Analysis: Separately analyzes the Face and Background of an image.
• Transformer Models: Uses ViT-B/16 (Vision Transformer) for high-accuracy feature extraction.
• Smart Thresholding:
  • Face Priority: Strict check on facial artifacts (Threshold: 0.4).
  • Background Check: Low sensitivity check for obvious background anomalies (Threshold: 0.05).
• Real-Time UI: Modern Next.js interface with instant feedback.

Architecture

The system employs a Twin-Stream Network approach:

graph TD
    A[Input Image] --> B{Segmentation}
    B -->|Extract| C[Face Region]
    B -->|Extract| D[Background Region]
    
    C --> E[Face Transformer ViT]
    D --> F[Background Transformer ViT]
    
    E --> G[Face Score]
    F --> H[Background Score]
    
    G --> I{Decision Logic}
    H --> I
    
    I -->|Face < 0.4| J[DEEPFAKE DETECTED]
    I -->|BG < 0.05| J
    I -->|Else| K[AUTHENTIC MEDIA]
    
    style J fill:#ff4d4d,stroke:#333,stroke-width:2px,color:white
    style K fill:#00cc66,stroke:#333,stroke-width:2px,color:white

Loading

Screenshots

Authentic Media	Deepfake Detected

Project Structure

• Frontend/: Next.js web application (The User Interface).
• Backend/: Flask API (The AI Engine).
• Backend/Model Files/: Contains the trained Keras models (face_transformer.keras, bg_transformer.keras).

Prerequisites

• Python 3.8+
• Node.js 18+

Installation

1. Backend Setup

cd Backend
pip install -r requirements.txt

2. Frontend Setup

cd Frontend/be_fr-master
npm install
# OR
pnpm install

How to Run

Option 1: One-Click (Recommended)

Simply double-click run_project.bat in the main folder.

• It automatically starts the Backend server.
• It starts the Frontend UI.
• It opens your default browser to the application.

Option 2: Manual Method

1. Start Backend:

cd Backend
python backend.py

2. Start Frontend:

cd Frontend/be_fr-master
npm run dev

Usage

1. Open the application (http://localhost:3000).
2. Upload an image (drag & drop or click to select).
3. The system will automatically:
   • Segment the Face and Background.
   • Analyze both using the Transformer models.
   • Display the result ("Authentic Media" or "Deepfake Detected").

Performance Metrics

The system has been evaluated on the DeepfakeTIMIT and Celeb-DF datasets.

Metric	Score
Face Model Accuracy	67.0%
Background Model Accuracy	83.0%
Combined System Accuracy	81.0%

Evaluation Charts

Accuracy Comparison	Confusion Matrix

Future Scope & Scalability

Since this project is a proof-of-concept, here is how it can be scaled and improved in the future:

• Cloud Deployment: Deploying the Flask backend to AWS Lambda/GCP for auto-scaling.
• Video Support: Extending the pipeline to process video files frame-by-frame.
• Browser Extension: Building a Chrome extension to automatically scan images on social media.
• Mobile App: Porting the UI to React Native for mobile deepfake detection.
• API Gateway: Creating a public API for other developers to use our detection engine.

License

This project is licensed under the MIT License - see the LICENSE file for details.