refactor cpg generation and query execution

Author: Lekssays

Dockerfile

@@ -1,32 +1,16 @@
-# Dockerfile for CodeBadger Toolkit Server
-# Container with Joern, Python MCP server, and Redis
+# Dockerfile for Joern Server Container
+# Contains Joern CLI and Redis for CPG generation and caching
 
 FROM eclipse-temurin:21-jdk-jammy
 
-# Install system dependencies including Python 3.12 and Redis
+# Install system dependencies
 RUN apt-get update && apt-get install -y \
     curl \
-    git \
     wget \
     unzip \
-    build-essential \
-    software-properties-common \
     redis-server \
-    libffi-dev \
-    libssl-dev \
-    && add-apt-repository ppa:deadsnakes/ppa \
-    && apt-get update \
-    && apt-get install -y \
-    python3.12 \
-    python3.12-venv \
-    python3.12-dev \
-    python3-pip \
     && rm -rf /var/lib/apt/lists/*
 
-# Set Python 3.12 as default
-RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.12 1 \
-    && update-alternatives --install /usr/bin/python python /usr/bin/python3.12 1
-
 # Set Joern version
 ENV JOERN_VERSION=4.0.429
 ENV JOERN_HOME=/opt/joern
@@ -43,22 +27,10 @@ RUN mkdir -p ${JOERN_HOME} && \
 # Add Joern CLI tools to PATH
 ENV PATH="${JOERN_HOME}/joern-cli:${JOERN_HOME}/joern-cli/bin:${PATH}"
 
-# Create workspace and playground directories
-RUN mkdir -p /workspace /playground /app
-
-# Copy MCP server code
-COPY . /app
-WORKDIR /app
+# Create playground directory for CPG storage
+RUN mkdir -p /playground
 
-# Install Python dependencies
-# Ensure pip/setuptools are available for python3 (pointing to 3.12)
-RUN python3 -m ensurepip --upgrade || true && \
-    python3 -m pip install --upgrade pip setuptools wheel && \
-    python3 -m pip install --no-cache-dir -r requirements.txt && \
-    python3 -m pip uninstall -y cryptography && \
-    python3 -m pip install --no-cache-dir cffi cryptography
-
-# Configure Redis to run in background
+# Configure Redis to run in background and listen on all interfaces
 RUN mkdir -p /var/lib/redis && \
     chown redis:redis /var/lib/redis && \
     sed -i 's/^daemonize no/daemonize yes/' /etc/redis/redis.conf && \
@@ -67,23 +39,19 @@ RUN mkdir -p /var/lib/redis && \
 # Verify Joern installation
 RUN joern --help
 
-# Expose MCP server port only
-EXPOSE 4242
+# Expose Redis port
+EXPOSE 6379
 
-# Create entrypoint script
+# Create entrypoint script that starts Redis
 RUN echo '#!/bin/bash\n\
 set -e\n\
 \n\
 # Start Redis in background\n\
 redis-server /etc/redis/redis.conf\n\
 \n\
-# Wait for Redis to be ready\n\
-sleep 2\n\
-\n\
-# Start MCP server\n\
-cd /app\n\
-exec python3 main.py\n\
-' > /app/entrypoint.sh && chmod +x /app/entrypoint.sh
+# Keep container running\n\
+tail -f /dev/null\n\
+' > /entrypoint.sh && chmod +x /entrypoint.sh
 
 # Run entrypoint script
-CMD ["/app/entrypoint.sh"]
+CMD ["/entrypoint.sh"]

README.md

@@ -2,22 +2,86 @@
 
 A containerized Model Context Protocol (MCP) server providing static code analysis using Joern's Code Property Graph (CPG) technology with support for Java, C/C++, JavaScript, Python, Go, Kotlin, C#, Ghidra, Jimple, PHP, Ruby, and Swift.
 
+## Prerequisites
+
+Before you begin, make sure you have:
+
+- **Docker** and **Docker Compose** installed
+- **Python 3.10+** (Python 3.13 recommended)
+- **pip** (Python package manager)
+
+To verify your setup:
+
+```bash
+docker --version
+docker-compose --version
+python --version
+```
+
 ## Quick Start
 
-### Build and Run the Container
+### 1. Install Python Dependencies
+
+```bash
+# Create a virtual environment (optional but recommended)
+python -m venv venv
+source venv/bin/activate  # On Windows: venv\Scripts\activate
+
+# Install dependencies
+pip install -r requirements.txt
+```
+
+### 2. Start the Docker Services (Joern + Redis)
 
 ```bash
-docker compose up --build
+docker compose up -d
+```
+
+This starts:
+- **Joern Server**: Static code analysis engine (runs CPG generation and queries)
+- **Redis**: Metadata storage (tracks codebases, ports, and CPG information)
+
+Verify services are running:
+
+```bash
+docker compose ps
+```
+
+### 3. Start the MCP Server
+
+```bash
+# Set the correct Redis port (maps to container's 6379)
+REDIS_PORT=6380 python main.py
 ```
 
 The MCP server will be available at `http://localhost:4242`.
 
-### Stop the Service
+### 4. Stop All Services
 
 ```bash
-docker compose down
+# Stop MCP server (Ctrl+C in terminal)
+
+# Stop Docker services
+docker-compose down
+
+# Optional: Clean up everything
+bash cleanup.sh
 ```
 
+## Cleanup Script
+
+Use the provided cleanup script to reset your environment:
+
+```bash
+bash cleanup.sh
+```
+
+This will:
+- Stop and remove Docker containers
+- Kill orphaned Joern/MCP processes
+- Clear Python cache (`__pycache__`, `.pytest_cache`)
+- Optionally clear the playground directory (CPGs and cached codebases)
+
 ## Integrations 
 
 ### GitHub Copilot Integration
@@ -100,49 +164,121 @@ Add the following:
 
 Thanks for contributing! Here's a quick guide to get started with running tests and contributing code.
 
-Prerequisites
+### Prerequisites
+
 - Python 3.10+ (3.13 is used in CI)
 - Docker and Docker Compose (for integration tests)
 
-Local development
+### Local Development Setup
+
 1. Create a virtual environment and install dependencies
 
 ```bash
 python -m venv venv
-source venv/bin/activate
 pip install -r requirements.txt
 ```
 
-2. Run unit tests
+2. Start Docker services (for integration tests)
+
+```bash
+docker-compose up -d
+```
+
+3. Run unit tests
+
+```bash
+pytest tests/ -q
+```
+
+4. Run integration tests (requires Docker Compose running)
 
 ```bash
-pytest -q
+# Start MCP server in background
+REDIS_PORT=6380 python main.py &
+
+# Run integration tests
+pytest tests/integration -q
+
+# Stop MCP server
+pkill -f "python main.py"
 ```
 
-3. Run integration tests (requires Docker Compose)
+5. Run all tests
 
 ```bash
-docker compose up --build -d
-pytest -q tests/integration
-docker compose down
+pytest tests/ -q
 ```
 
-4. Run all tests
+6. Cleanup after testing
 
 ```bash
-pytest -q
+bash cleanup.sh
+docker-compose down
 ```
 
-Please follow the repository conventions and open a PR with a clear changelog and tests for changes that affect behavior.
+### Code Contributions
+
+Please follow these guidelines when contributing:
+
+1. Follow repository conventions
+2. Write tests for behavioral changes
+3. Ensure all tests pass before submitting PR
+4. Include a clear changelog in your PR description
+5. Update documentation if needed
 
 ## Configuration
 
-Optional configuration via `config.yaml` (copy from `config.example.yaml`).
+The MCP server can be configured via environment variables or `config.yaml`.
+
+### Environment Variables
+
+Key settings (optional - defaults shown):
 
-Key settings:
-- Server host/port
-- Redis settings
-- Session timeouts
-- CPG generation settings
+```bash
+# Server
+MCP_HOST=0.0.0.0
+MCP_PORT=4242
+
+# Redis (running inside Docker container)
+REDIS_HOST=localhost
+REDIS_PORT=6380        # ⚠️  IMPORTANT: Port 6380 on host maps to 6379 in container
+
+# Joern
+JOERN_BINARY_PATH=joern
+JOERN_JAVA_OPTS="-Xmx4G -Xms2G -XX:+UseG1GC -Dfile.encoding=UTF-8"
+
+# CPG Generation
+CPG_GENERATION_TIMEOUT=600
+MAX_REPO_SIZE_MB=500
+
+# Query
+QUERY_TIMEOUT=30
+QUERY_CACHE_ENABLED=true
+QUERY_CACHE_TTL=300
+```
+
+### Config File
+
+Create a `config.yaml` from `config.example.yaml`:
+
+```bash
+cp config.example.yaml config.yaml
+```
+
+Then customize as needed.
+
+### Important: Redis Port Configuration
+
+Since Redis runs inside the Docker container:
+
+- **Inside container**: Redis listens on `6379`
+- **Host mapping**: Docker maps `6380:6379`
+- **MCP server should use**: `REDIS_PORT=6380`
+
+Always start the MCP server with:
+
+```bash
+REDIS_PORT=6380 python main.py
+```
 
 

config.example.yaml

@@ -1,11 +1,11 @@
 server:
-  host: ${MCP_HOST:0.0.0.0}
+  host: ${MCP_HOST:127.0.0.1}
   port: ${MCP_PORT:4242}
   log_level: ${MCP_LOG_LEVEL:INFO}
 
 redis:
   host: ${REDIS_HOST:localhost}
-  port: ${REDIS_PORT:6379}
+  port: ${REDIS_PORT:6380}
   password: ${REDIS_PASSWORD:}
   db: ${REDIS_DB:0}
 
@@ -478,3 +478,4 @@ query:
 storage:
   workspace_root: ${WORKSPACE_ROOT:/tmp/codebadger-toolkit}
   cleanup_on_shutdown: ${CLEANUP_ON_SHUTDOWN:true}
+

docker-compose.yml

@@ -1,27 +1,25 @@
 version: '3.8'
 
 services:
-  codebadger-toolkit:
+  # Joern server with Redis (container)
+  codebadger-joern-server:
     build:
       context: .
       dockerfile: Dockerfile
-    image: codebadger-toolkit:latest
-    container_name: codebadger-toolkit-server
+    image: codebadger-joern-server:latest
+    container_name: codebadger-joern-server
     ports:
-      - "4242:4242"
-    environment:
-      - MCP_HOST=0.0.0.0
-      - MCP_PORT=4242
-      - REDIS_HOST=localhost
-      - REDIS_PORT=6379
-      - PYTHONUNBUFFERED=1
+      - "6380:6379"
+      # Expose ports 2000-2999 for Joern servers
+      - "2000-2999:2000-2999"
     volumes:
-      # Mount playground for persistent CPG storage
-      - ./playground:/app/playground
+      # Mount playground for CPG storage (shared with host)
+      - ./playground:/playground
     restart: unless-stopped
     healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:4242/mcp"]
+      test: ["CMD", "redis-cli", "ping"]
       interval: 10s
       timeout: 5s
       retries: 3
-      start_period: 30s
+      start_period: 10s
+