Improve Admin Area - Enhanced Search and Indexing Logs

[Dawe000]

Improve Admin Area - Enhanced Search and Indexing Logs

Description

Enhance the admin area's search logs and indexing logs pages with more detailed information when logs are expanded. This requires updates to the worker's logging, database schema, and admin UI to provide better visibility into system operations and debugging capabilities.

Requirements

• Enhanced Search Logs (when expanded):
  • Show full query details (not truncated)
  • Display complete filters object (formatted JSON, collapsible)
  • Show request headers (if available)
  • Display full error message and stack trace (if error occurred)
  • Show response metadata (if available)
  • Display vector search details (similarity scores, if available)
  • Show agent IDs returned (expandable list)
  • Display timing breakdown (query time, vector search time, embedding time, etc.)
  • Show request ID
  • Copy functionality for query, filters, error messages
• Enhanced Indexing Logs (when expanded):
  • Show detailed chain-by-chain breakdown:
    • Per-chain statistics (agents indexed, deleted, batches)
    • Per-chain duration
    • Per-chain status
    • Per-chain error messages (if any)
  • Display detailed event timeline:
    • Chronological list of all events
    • Event details (batch size, agent IDs, timestamps)
    • Visual timeline/gantt chart (optional)
    • Filter events by type (batch-processed, no-op, error, chain-complete)
    • Filter events by chain
  • Show agent IDs indexed/deleted (expandable lists, searchable)
  • Display configuration used (batch sizes, chains, etc.)
  • Show queue status and processing details
  • Display full error stack traces (if errors occurred)
  • Copy functionality for error messages, agent IDs, etc.

Worker Changes Required

• Request Logging Enhancements:
  • Log request headers (user-agent, referer, etc.)
  • Log response metadata (total results available, pagination info)
  • Log vector search details (similarity scores, embedding time)
  • Log agent IDs returned in search results
  • Log timing breakdown (embedding time, vector search time, post-processing time)
  • Log request ID
• Indexing Logging Enhancements:
  • Log batch size per event
  • Log processing time per batch
  • Log configuration used (batch sizes, chains, etc.)
  • Log queue processing details
  • Log more detailed error information (stack traces, context)

Database Schema Changes Required

• request_logs table additions:
  • request_id TEXT (for request tracking)
  • request_headers TEXT (JSON string of headers)
  • response_metadata TEXT (JSON string of response metadata)
  • vector_search_details TEXT (JSON string with similarity scores, embedding time, etc.)
  • agent_ids_returned TEXT (JSON array of agent IDs)
  • timing_breakdown TEXT (JSON object with timing details)
  • error_stack TEXT (full error stack trace)
• sync_log_events table additions:
  • batch_size INTEGER (size of batch processed)
  • processing_time_ms INTEGER (time taken to process this batch)
  • configuration TEXT (JSON string of configuration used)
  • error_stack TEXT (full error stack trace if error)
• sync_logs table additions (optional):
  • configuration TEXT (JSON string of configuration used for this sync)
  • queue_status TEXT (JSON string with queue processing details)

UI Improvements

• Better visual hierarchy for expanded details
• Collapsible sections for large data (agent IDs, filters, etc.)
• Syntax highlighting for JSON data
• Better formatting for timestamps (relative + absolute)
• Search/filter within expanded details
• Export functionality (export log details as JSON)
• Visual timeline for indexing events

Technical Notes

• Create new migration file for schema changes
• Update RequestLogger class to log additional fields
• Update SyncEventLogger class to log additional fields
• Update SyncLogger class if needed
• Update search handler to capture and log additional data
• Update queue processor to log additional details
• Update API endpoints to return new fields:
  • /api/admin/logs/search - return new request log fields
  • /api/admin/logs/indexing - return new sync log fields
  • /api/admin/logs/indexing/[id]/events - return new event fields
• Use existing component library (shadcn/ui) for consistency
• Consider using a JSON viewer component for formatted JSON display

Potential Sub-issues

• Database schema migration for enhanced logging
• Worker logging enhancements (request and indexing)
• Enhanced search log detail view
• Enhanced indexing log detail view with chain breakdown
• Event timeline visualization
• Agent ID lists with search/filter
• JSON viewer and formatting improvements