LogEngine - Intelligent Log Classification System

An AI-powered log classification engine that intelligently categorizes system logs using a hybrid log classification system, combining three complementary approaches to handle varying levels of complexity in log patterns. The classification methods ensure flexibility and effectiveness in processing predictable, complex, and poorly-labeled data patterns.

Features • Quick Start • Architecture • API Documentation • Contributing

---

Table of Contents

1. Overview
2. Features
3. System Architecture
4. Installation
5. Quick Start
6. API Documentation
7. Configuration
8. Development
9. Contributing
10. License

---

Overview

LogEngine is an intelligent log classification system designed to automatically categorize system logs into predefined categories using a hybrid approach combining regex-based pattern matching, BERT-based semantic analysis, and Large Language Models (LLM).

The system processes CSV files containing log messages from multiple sources (e.g., CRM systems, billing systems, analytics engines) and produces enriched output with intelligent classification labels. This enables organizations to:

• Automate log categorization at scale
• Reduce manual review time by up to 80%
• Identify critical issues faster through semantic understanding
• Support legacy systems with specialized LLM processing
• Handle edge cases with fallback mechanisms

Key Use Cases

• Enterprise Log Management: Automatically categorize logs from distributed systems
• Security Monitoring: Identify anomalies and security-related events
• Incident Response: Fast-track critical issues for immediate action
• Compliance Reporting: Categorize logs for audit and compliance purposes
• System Health Monitoring: Track system notifications and operational events

---

Features

🎯 Multi-Algorithm Classification

• Regex-Based Matching: Ultra-fast pattern recognition for known log formats
• BERT Embeddings: Deep semantic understanding for complex log messages
• LLM Integration: Advanced reasoning for ambiguous or legacy system logs
• Intelligent Fallback: Graceful degradation when primary method fails

📊 Flexible Output

• CSV Processing: Batch process multiple logs in one operation
• RESTful API: Real-time classification via FastAPI
• Extensible Categories: Easily add new classification categories
• Confidence Scoring: Built-in confidence metrics (via BERT probabilities)

🚀 Performance Optimized

• Pre-trained Models: Uses lightweight sentence transformers for fast inference
• Efficient Caching: Model loading optimization
• Scalable Design: Built on FastAPI for async processing
• Low Latency: < 100ms per log message on average

🔒 Production Ready

• Error Handling: Comprehensive exception management
• Input Validation: CSV schema validation
• Logging: Detailed processing logs for debugging
• Resource Efficient: Minimal memory footprint

---

System Architecture

Installation

Prerequisites

• Python 3.9+
• pip or conda
• Groq API Key (for LLM features)
• 8GB RAM minimum (for BERT model loading)

Step 1: Clone Repository

git clone https://github.com/yourusername/logengine.git
cd logengine

Step 2: Create Virtual Environment

# Using venv
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Or using conda
conda create -n logengine python=3.9
conda activate logengine

Step 3: Install Dependencies

pip install -r requirements.txt

Step 4: Download Pre-trained Models

# BERT models will auto-download on first run
# Alternatively, pre-download:
python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('all-MiniLM-L6-v2')"

Step 5: Configure Environment Variables

# Create .env file
cat > .env << EOF
GROQ_API_KEY=your_groq_api_key_here
LOG_LEVEL=INFO
EOF

Step 6: Verify Installation

# Test imports
python -c "from classify import classify; print('✓ LogEngine installed successfully')"

# Test API server
python server.py
# Should output: INFO:     Uvicorn running on http://0.0.0.0:8000

---

Quick Start

Option 1: Using the REST API

# Start the server
python server.py

# In another terminal, upload a CSV file
curl -X POST "http://localhost:8000/classify/" \
  -F "file=@resources/test.csv"

# Response: CSV file with target_label column

Option 2: Batch Processing (Script)

# Classify logs from a CSV file
python classify.py

# Input: resources/test.csv
# Output: resources/output.csv

Option 3: Python API

from classify import classify_log, classify

# Single log classification
label = classify_log("ModernCRM", "User 12345 logged in.")
print(label)  # Output: "User Action"

# Batch classification
logs = [
    ("ModernCRM", "User 12345 logged in."),
    ("BillingSystem", "Payment received successfully"),
    ("LegacyCRM", "The ReportGenerator module will be retired in v4.0")
]
labels = classify(logs)
for log, label in zip(logs, labels):
    print(f"{log[0]} → {label}")

---

API Documentation

POST /classify/

Classify log messages from a CSV file

Request

POST /classify/ HTTP/1.1
Host: localhost:8000
Content-Type: multipart/form-data

file: (binary CSV file)

CSV Input Format

source,log_message
ModernCRM,User 12345 logged in.
BillingSystem,Payment received for invoice INV-001
AnalyticsEngine,File data_6957.csv uploaded successfully by user User265
LegacyCRM,The ReportGenerator module will be retired in version 4.0

Response (Success)

HTTP/1.1 200 OK
Content-Type: text/csv

source,log_message,target_label
ModernCRM,User 12345 logged in.,User Action
BillingSystem,Payment received for invoice INV-001,System Notification
AnalyticsEngine,File data_6957.csv uploaded successfully by user User265,System Notification
LegacyCRM,The ReportGenerator module will be retired in version 4.0,Deprecation Warning

Response (Error)

{
  "detail": "CSV must contain 'source' and 'log_message' columns."
}

Status Codes

Code	Description
200	Classification successful
400	Invalid file format or missing columns
500	Server error during processing

---

Configuration

Environment Variables

# Required
GROQ_API_KEY=sk-grok_xxxxxxxxxxxxx

# Optional
LOG_LEVEL=INFO                 # DEBUG, INFO, WARNING, ERROR
CONFIDENCE_THRESHOLD=0.5       # BERT confidence threshold
API_HOST=0.0.0.0              # FastAPI host
API_PORT=8000                 # FastAPI port

Model Configuration

Edit the processor files to customize models:

# processor_bert.py
model_embedding = SentenceTransformer('all-MiniLM-L6-v2')

# processor_llm.py
model="deepseek-r1-distill-llama-70b"
temperature=0.5

Adding Custom Regex Patterns

Edit processor_regex.py:

regex_patterns = {
    r"Your pattern here": "Your Category",
    r"Another pattern": "Another Category",
}

---

Development

Project Structure

logengine/
├── server.py                  # FastAPI server
├── classify.py               # Classification orchestrator
├── processor_regex.py        # Regex-based processor
├── processor_bert.py         # BERT-based processor
├── processor_llm.py          # LLM-based processor
├── models/
│   ├── log_classifier.joblib # Pre-trained BERT classifier
│   └── log_classifier_model.pkl # Backup model
├── resources/
│   ├── test.csv             # Test dataset
│   └── output.csv           # Output results
├── .env                      # Environment variables
├── requirements.txt          # Python dependencies
└── README.md                 # This file

Running Tests

# Unit tests
python -m pytest tests/ -v

# Integration tests
python -m pytest tests/integration/ -v

# Test classification on sample data
python classify.py

Performance Profiling

# Profile classification speed
python -m cProfile -s cumtime classify.py

# Memory usage
python -m memory_profiler classify.py

Debugging

# Add debug logging
import logging
logging.basicConfig(level=logging.DEBUG)

# Log classification steps
from classify import classify_log
label = classify_log("ModernCRM", "User 12345 logged in.", debug=True)

---

Contributing

We welcome contributions! Please follow these guidelines:

Process

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Write tests for new functionality
5. Commit changes (git commit -m 'Add amazing feature')
6. Push to branch (git push origin feature/amazing-feature)
7. Open a Pull Request

Coding Standards

• Follow PEP 8 guidelines
• Add docstrings to all functions
• Include type hints
• Write unit tests (minimum 80% coverage)
• Update README for new features

Testing Requirements

# Must pass all tests
pytest tests/ --cov=. --cov-report=term-missing

# Code quality
flake8 . --max-line-length=100
black . --check

---

Troubleshooting

Common Issues

Issue: "ModuleNotFoundError: No module named 'groq'"

# Solution: Install missing dependency
pip install groq python-dotenv

Issue: "GROQ_API_KEY environment variable not found"

# Solution: Create .env file with valid key
echo "GROQ_API_KEY=your_key_here" > .env

Issue: BERT model slow to load

# Solution: Model downloads on first use, subsequent runs are fast
# To pre-download: python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('all-MiniLM-L6-v2')"

Issue: CSV Processing fails

# Solution: Ensure CSV has required columns
# Required: source, log_message
# Check file encoding (should be UTF-8)

Performance Tips

1. Batch Processing: Use script mode for 100+ logs
2. Model Caching: First run may be slow (model download)
3. API Optimization: Use connection pooling for multiple requests
4. GPU Support: BERT supports CUDA for faster inference

---

License

This project is licensed under the MIT License - see LICENSE file for details.

---

Support & Resources

• Documentation: Full API Docs
• Examples: Example Scripts
• Issues: GitHub Issues
• Discussions: GitHub Discussions

---

Acknowledgments

• Groq for providing fast LLM inference
• Hugging Face for sentence transformers
• Scikit-learn for ML classification
• FastAPI for the web framework

---

Contact

If you have any questions or feedback, please feel free to contact me at pranjal360agarwal@gmail.com. You can also connect with me on LinkedIn or Twitter. Thank you for visiting my project!

Made with ❤ by Pranjal Agarwal.