Implement AST-based semantic chunking as default behavior (v2.0.0.0)

BREAKING CHANGES:
- Removed deprecated legacy indexing methods from GitAwareDocumentProcessor
- AST-based semantic chunking is now the default for all indexing operations
- Removed index_codebase() and update_index_smart() methods

Major features:
- AST-based semantic chunking for Python, JavaScript, TypeScript, Java, and Go
- Semantic metadata extraction (functions, classes, methods with signatures)
- Enhanced search with semantic filters (--type, --scope, --features, --parent)
- Intelligent chunk size management with semantic boundary preservation
- Fallback to text chunking for unsupported languages or malformed code

Infrastructure improvements:
- Fixed test isolation issues causing E2E test failures
- Enhanced podman recovery script for stuck containers
- Improved test setup order (services must start before indexing)
- Fixed timing constraints in service readiness tests
- Added proper E2E test markers to exclude from CI

Query enhancements:
- Display semantic metadata in search results (type, name, signature)
- Support for semantic-only filtering to exclude text chunks
- Show AST-extracted attributes in verbose mode
- Maintain line number display with semantic chunks

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

Author: jsbattig

README.md

@@ -4,11 +4,13 @@ AI-powered semantic code search for your codebase. Find code by meaning, not jus
 
 ## Features
 
-- **Semantic Search** - Find code by meaning using vector embeddings
+- **Semantic Search** - Find code by meaning using vector embeddings and AST-based semantic chunking
 - **Multiple Providers** - Local (Ollama) or cloud (VoyageAI) embeddings  
 - **Smart Indexing** - Incremental updates, git-aware, multi-project support
+- **Semantic Filtering** - Filter by code constructs (classes, functions), scope, language features
+- **Multi-Language Support** - AST parsing for Python, JavaScript, TypeScript, Java, Go
 - **CLI Interface** - Simple commands with progress indicators
-- **AI Analysis** - Integrates with Claude CLI for code analysis
+- **AI Analysis** - Integrates with Claude CLI for code analysis with semantic search
 - **Privacy Options** - Full local processing or cloud for better performance
 
 ## Installation
@@ -38,6 +40,10 @@ code-indexer index
 # Search semantically
 code-indexer query "authentication logic"
 
+# Search with semantic filtering
+code-indexer query "user" --type class --scope global
+code-indexer query "save" --features async --language python
+
 # AI-powered analysis (requires Claude CLI)
 code-indexer claude "How does auth work in this app?"
 ```
@@ -55,7 +61,9 @@ code-indexer stop                     # Stop services
 
 # Additional options
 code-indexer index --clear            # Force full reindex
+code-indexer index --reconcile        # Reconcile disk vs database
 code-indexer query "auth" --limit 20  # More results
+code-indexer query "function" --type function --semantic-only  # Semantic filtering
 code-indexer watch                    # Real-time updates
 cidx query "search"                   # Short alias
 ```
@@ -79,18 +87,21 @@ code-indexer init --embedding-provider voyage-ai
 
 During indexing, VoyageAI shows real-time performance status in the progress bar:
 - ⚡ **Full speed** - Running at maximum throughput
+- 🟡 **CIDX throttling** - Internal rate limiter active
 - 🔴 **Server throttling** - VoyageAI API rate limits detected, automatically backing off
 
 Example: `15/100 files (15%) | 8.3 emb/s ⚡ | 8 threads | main.py`
 
-The system runs at full speed by default and only backs off when the API server enforces rate limits.
+The system runs at full speed by default and only backs off when rate limits are encountered.
 
 ### Configuration File
 Configuration is stored in `.code-indexer/config.json`:
 - `file_extensions`: File types to index
 - `exclude_dirs`: Directories to skip  
 - `chunk_size`: Text chunk size
 - `embedding_provider`: ollama or voyage-ai
+- `use_semantic_chunking`: Enable AST-based semantic chunking (default: true)
+- `max_file_size`: Maximum file size in bytes (default: 1MB)
 
 ## Requirements
 

RELEASE_NOTES.md

@@ -1,5 +1,63 @@
 # Code Indexer Release Notes
 
+## Version 2.0.0.0 (2025-01-11)
+
+### 🚨 BREAKING CHANGES
+
+#### **Legacy Indexing Methods Removed**
+- **Removed deprecated methods**: Completely removed `index_codebase()` and `update_index_smart()` from `GitAwareDocumentProcessor`
+- **Removed deprecated methods**: Completely removed `index_codebase()` and `update_index()` from `DocumentProcessor`
+- **API Breaking Change**: These methods now only exist in `SmartIndexer` which is the single source of truth for all indexing operations
+- **Migration Required**: Any external code calling these deprecated methods must be updated to use `SmartIndexer.smart_index()` instead
+
+#### **AST-Based Semantic Chunking is Now Default**
+- **Configuration Change**: `use_semantic_chunking` now defaults to `True` instead of `False`
+- **Enhanced Search Results**: All new indexes will use AST-based semantic chunking by default for improved code understanding
+- **Fallback Behavior**: Automatically falls back to text chunking for unsupported languages or malformed code
+
+### 🚀 Major Features & Enhancements
+
+#### **Codebase Architecture Cleanup**
+- **Simplified Indexing Paths**: Consolidated all indexing operations through `SmartIndexer` eliminating code duplication
+- **Removed Dead Code**: Eliminated unreachable indexing methods that were no longer used in production
+- **Cleaner Architecture**: Streamlined processor hierarchy with clear separation of concerns
+
+#### **Enhanced Test Infrastructure**
+- **Container Reuse Optimization**: Fixed test infrastructure to reuse containers between tests instead of creating new ones
+- **Improved E2E Tests**: AST chunking E2E tests now use shared project directories and `index --clear` for data reset
+- **Better Test Performance**: Reduced container creation overhead in test suite by using proper shared infrastructure
+
+#### **Code Quality Improvements**
+- **Full Linting Compliance**: All code now passes ruff, black, and mypy checks
+- **Type Safety**: Enhanced type checking across all modified modules
+- **Documentation Updates**: Updated method signatures and documentation to reflect current architecture
+
+### 🐛 Bug Fixes
+- **Container Proliferation**: Fixed issue where E2E tests were creating new containers for each test run
+- **Test Infrastructure**: Fixed project hash calculation issues in test environment
+- **Semantic Metadata**: Fixed semantic metadata storage and display in query results
+
+### 🔧 Technical Improvements
+- **Reduced Code Complexity**: Removed approximately 100 lines of deprecated code across processor classes
+- **Better Error Messages**: Deprecated methods now provide clear guidance on replacements
+- **Consistent API**: All indexing now goes through the same well-tested code path
+
+### 📊 Breaking Change Impact Analysis
+- **External API Users**: Any code directly calling `processor.index_codebase()` or `processor.update_index_smart()` must migrate
+- **CLI Users**: No impact - all CLI commands continue to work unchanged
+- **SmartIndexer Usage**: Recommended migration path is to use `SmartIndexer.smart_index()` for all indexing operations
+
+### 🏗️ Migration Guide
+```python
+# OLD (no longer works)
+processor = GitAwareDocumentProcessor(config, embedding_provider, qdrant_client)
+stats = processor.index_codebase(clear_existing=True)
+
+# NEW (recommended)
+smart_indexer = SmartIndexer(config, embedding_provider, qdrant_client)
+stats = smart_indexer.smart_index(clear_existing=True)
+```
+
 ## Version 1.1.0.0 (2025-01-05)
 
 ### 🚀 Major Feature: Copy-on-Write (CoW) Clone Support

ci-github.sh

@@ -91,6 +91,7 @@ echo "ℹ️  This matches GitHub Actions - only unit tests that don't require e
 
 # Run the exact same test command as GitHub Actions
 if PYTHONPATH="$(pwd)/src:$(pwd)/tests" pytest tests/ \
+    -m "not e2e" \
     --ignore=tests/test_e2e_embedding_providers.py \
     --ignore=tests/test_start_stop_e2e.py \
     --ignore=tests/test_end_to_end_complete.py \
@@ -126,6 +127,10 @@ if PYTHONPATH="$(pwd)/src:$(pwd)/tests" pytest tests/ \
     --ignore=tests/test_indexing_consistency_e2e.py \
     --ignore=tests/test_timestamp_comparison_e2e.py \
     --ignore=tests/test_line_number_display_e2e.py \
+    --ignore=tests/test_semantic_query_display_e2e.py \
+    --ignore=tests/test_semantic_search_capabilities_e2e.py \
+    --ignore=tests/test_semantic_chunking_ast_fallback_e2e.py \
+    --ignore=tests/test_cancellation_high_throughput_processor.py \
     --ignore=tests/test_concurrent_indexing_prevention.py \
     --ignore=tests/test_resume_and_incremental_bugs.py \
     --ignore=tests/test_actual_file_chunking.py \

fix-podman-stuck.sh

@@ -106,9 +106,13 @@ reset_podman_completely() {
     echo "  Stopping all podman user services..."
     systemctl --user stop podman.socket podman.service 2>/dev/null || true
 
-    # Kill any remaining podman processes
+    # Kill any remaining podman processes more aggressively
     echo "  Killing any remaining podman processes..."
     pkill -f podman 2>/dev/null || echo "    No podman processes to kill"
+    pkill -9 podman 2>/dev/null || true
+    pkill -9 conmon 2>/dev/null || true
+    pkill -9 crun 2>/dev/null || true
+    pkill -9 runc 2>/dev/null || true
 
     # Wait for processes to die
     sleep 5
@@ -117,15 +121,34 @@ reset_podman_completely() {
     echo "  Cleaning runtime directories..."
     rm -rf ~/.local/share/containers/storage/tmp/* 2>/dev/null || true
     rm -rf /run/user/$(id -u)/containers/* 2>/dev/null || true
+    rm -rf /run/user/$(id -u)/libpod/* 2>/dev/null || true
+    rm -rf /tmp/podman-run-$(id -u)/ 2>/dev/null || true
+    rm -rf /tmp/containers-user-$(id -u)/ 2>/dev/null || true
 
-    # Reset podman system
-    echo "  Resetting podman system..."
-    timeout 30s podman system reset --force 2>/dev/null || echo "    System reset failed"
+    # If podman is completely stuck, remove the entire storage
+    echo "  Removing podman storage (complete reset)..."
+    rm -rf ~/.local/share/containers/ 2>/dev/null || true
+    rm -rf ~/.config/containers/ 2>/dev/null || true
+    
+    # Reset podman system (might fail if podman is stuck)
+    echo "  Attempting podman system reset..."
+    timeout 10s podman system reset --force 2>/dev/null || echo "    System reset failed - proceeding with manual cleanup"
+    
+    # Reinitialize podman
+    echo "  Reinitializing podman..."
+    systemctl --user daemon-reload
+    
+    # Migrate podman to create fresh directories
+    echo "  Running podman system migrate..."
+    podman system migrate 2>/dev/null || echo "    Migration failed - will retry after service start"
 
     # Restart podman
     echo "  Restarting podman service..."
     systemctl --user start podman.socket
     sleep 5
+    
+    # Try migration again after service start
+    podman system migrate 2>/dev/null || true
 }
 
 # Function to clean system-wide if needed (requires sudo)
@@ -136,6 +159,20 @@ system_wide_cleanup() {
     echo "  Cleaning system-wide container resources..."
     sudo systemctl stop podman.socket podman.service 2>/dev/null || true
     sudo pkill -f podman 2>/dev/null || echo "    No system podman processes to kill"
+    sudo pkill -9 podman 2>/dev/null || true
+    sudo pkill -9 conmon 2>/dev/null || true
+    
+    # Clean up any hanging mounts
+    echo "  Cleaning up hanging mounts..."
+    sudo umount -f /run/user/$(id -u)/netns/* 2>/dev/null || true
+    sudo umount -f /var/lib/containers/storage/overlay/* 2>/dev/null || true
+    sudo umount -f /run/containers/storage/* 2>/dev/null || true
+    
+    # Remove system-wide podman directories if they exist
+    echo "  Removing system podman directories..."
+    sudo rm -rf /var/lib/containers/ 2>/dev/null || true
+    sudo rm -rf /run/containers/ 2>/dev/null || true
+    sudo rm -rf /run/libpod/ 2>/dev/null || true
 
     # Clean up cgroup resources
     echo "  Cleaning cgroup resources..."
@@ -144,17 +181,27 @@ system_wide_cleanup() {
         sudo rmdir "$dir" 2>/dev/null || true
     done
 
-    # Clean up network namespaces
+    # Clean up network namespaces more aggressively
     echo "  Cleaning network namespaces..."
-    sudo ip netns list 2>/dev/null | grep -E "netns|cni" | while read ns rest; do
+    sudo ip netns list 2>/dev/null | grep -E "netns|cni|podman" | awk '{print $1}' | while read ns; do
         echo "    Removing network namespace: $ns"
         sudo ip netns delete "$ns" 2>/dev/null || true
     done
 
+    # Clean up any CNI networks
+    echo "  Cleaning CNI networks..."
+    sudo rm -rf /var/lib/cni/ 2>/dev/null || true
+    sudo rm -rf /etc/cni/net.d/ 2>/dev/null || true
+    
     # Restart networking
     echo "  Restarting networking..."
     sudo systemctl restart NetworkManager 2>/dev/null || true
 
+    # Reload systemd
+    echo "  Reloading systemd..."
+    systemctl --user daemon-reload
+    sudo systemctl daemon-reload
+    
     # Start user podman again
     echo "  Starting user podman service..."
     systemctl --user start podman.socket

src/code_indexer/__init__.py

@@ -5,5 +5,5 @@
 to provide code search capabilities.
 """
 
-__version__ = "1.1.0.0"
+__version__ = "2.0.0.0"
 __author__ = "Code Indexer Team"