Advanced Logging Service & Command-Line Interface

[MrScripty]

1. Overview

This document outlines the complete architectural plan and implementation roadmap for creating a robust, decoupled logging service and a feature-rich, interactive Command-Line Interface (CLI) for whip_ui.

This plan is the final synthesis of our design discussions. It incorporates an analysis of the existing codebase, details the new architecture to be built, plans for the safe removal of obsolete code, and lays the groundwork for future diagnostic features. It is designed to be followed in incremental, testable steps, with a specific focus on delivering a functional, text-based CLI for logging and control before integrating an advanced UI.

2. Core Philosophy & Design Principles

The architecture of this system is guided by several core principles essential for building a maintainable, performant, and reliable developer tool.

• Hybrid Push/Pull Logging Architecture: The system will use a hybrid model to achieve both high performance and maximum flexibility.

  • Reasoning: A pure "push" system (where every consumer gets every log) can be inefficient, while a pure "pull" system (where consumers poll for logs) can be complex to implement performantly. This hybrid model gets the best of both.
  • Push (Performance): A global, performant tracing_subscriber::EnvFilter acts as the first gate for all log events. This is a "push" mechanism that runs before any significant processing. Its primary purpose is to discard high-volume logs (like trace!) as early as possible, ensuring the application has near-zero logging overhead during normal operation or in release builds.
  • Pull (Flexibility): All logs that pass the global filter are processed and placed into a central, in-memory CentralLogStore. This store acts as a temporary, circular database of recent log activity. Any consumer (like the CLI) can then access this central store and "pull" the data it needs, applying its own secondary filters for display. This allows the CLI to show info logs while another future tool could simultaneously show only diagnostics logs, all from the same data source.

• Strict Separation of Concerns (Service vs. Consumer): The logging service is a foundational, backend component, completely independent of any log consumer.

  • Reasoning: This is the most critical principle for long-term maintainability. By decoupling the "what" (the log data) from the "how" (the display), we can add, remove, or change consumers without ever touching the core logging pipeline.
  • The Logging Service: Manages the global filter, processes incoming logs (e.g., for deduplication), and populates the CentralLogStore. It has no knowledge of how the logs will be displayed.
  • Log Consumers: The cli module is one such consumer. It reads from the CentralLogStore and is solely responsible for presentation and user interaction. This separation means a future in-game GUI console, a network log streamer, or a file logger could be added as new consumers without altering the existing CLI.

• Concurrency Safety & Performance: A desktop application must remain responsive at all times.

  • Reasoning: A diagnostic tool must not interfere with the performance of the application it's diagnosing.
  • The Command-Line Interface will run in its own dedicated thread to prevent its I/O operations (reading user input, drawing to the terminal) from blocking the main application's render loop.
  • The CentralLogStore will be shared between threads safely using an Arc<Mutex<...>>. Consumers will be designed to lock the mutex, clone the data they need, and unlock it as quickly as possible to minimize contention.
  • Communication from the CLI thread to the main Bevy App will use bounded crossbeam-channels to prevent unbounded memory growth if the main thread is busy.

• Graceful Degradation & Fallbacks: The diagnostic tools must be the most reliable part of the application.

  • Reasoning: A debugger that crashes when the main application has a problem is useless.
  • The CLI will be built with a simple, robust, text-only renderer first. The advanced, layout-driven UI will be an enhancement that can gracefully fall back to the basic renderer if its assets (e.g., layout files) fail to load or are invalid.
  • The CLI thread will use a Drop guard to ensure that, even in the event of a panic, the user's terminal is restored to a usable state.

• Scoped Error Handling: The system's response to errors will be appropriate to the error's severity.

  • Fail-Fast on Core Initialization: The application will terminate immediately if a critical, non-recoverable error occurs during core setup (e.g., failure to enter terminal raw mode). This prevents the application from running in a broken state.
  • Graceful In-Loop Error Handling: Recoverable errors that occur during operation (e.g., a user typing an invalid command into the CLI) will be handled gracefully, displaying a status message to the user without crashing the application.

---

3. Phased Implementation Plan

Phase 0: Prerequisite Work (Already Completed)

• Objective: To establish a sane, semantic baseline for all log messages across the application, reducing noise from per-frame events.
• Actionable Steps (Completed):
  1. Log Level Restructuring: All logging statements in the codebase were updated to use appropriate levels (ERROR, WARN, INFO, DEBUG, TRACE).
  2. Specific Changes Made: rendering_system logs moved to trace!, debug_shape_visibility_system logs moved to debug!, etc.

Phase 1: Project Setup & New Architecture Foundation

• Objective: Prepare the project structure for the new architecture and add all necessary dependencies.

• Actionable Steps:

  1. Update whip_ui/Cargo.toml with the following dependencies.

     [dependencies]
     # ... existing dependencies
     crossterm = "0.27"
     ratatui = { version = "0.26", features = ["crossterm"] }
     tracing = "0.1"
     tracing-subscriber = { version = "0.3", features = ["env-filter", "reload", "registry", "fmt"] }
     anyhow = "1.0"
     crossbeam-channel = "0.5"
     # serde, serde_json, thiserror are already present

  2. Create the decoupled directory and file structure:

     whip_ui/src/
     ├── logging/
     │   ├── mod.rs
     │   └── structs.rs
     ├── cli/
     │   ├── mod.rs
     │   ├── plugin.rs
     │   ├── renderer.rs
     │   ├── error.rs
     │   ├── commands.rs
     │   └── session.rs  // For the terminal Drop guard
     └── ... (other modules)
     

• ✅ Testable Outcome: The project compiles successfully with the new dependencies and module structure.

Phase 2: Core Logging Service & Central Store

• Objective: Build the independent logging infrastructure, including the central in-memory log store, processing layers, and a formal configuration structure. This will replace Bevy's default LogPlugin.

• Actionable Steps:

  1. Define Core Data Structures and Config in logging/structs.rs:

     use bevy_ecs::prelude::Resource;
     use std::sync::{Arc, Mutex};
     use std::collections::VecDeque;
     use tracing::{Level, span};
     use std::time::SystemTime;
     use tracing_subscriber::{reload, EnvFilter, Registry};
     
     #[derive(Resource, Clone)]
     pub struct LoggingConfig {
         pub central_store_capacity: usize,
         pub default_global_filter: String,
     }
     
     #[derive(Clone, Debug)]
     pub struct ProcessedLog {
         pub id: u64, // A unique, incrementing ID for each log
         pub timestamp: SystemTime,
         pub level: Level,
         pub target: String, // e.g., "whip_ui::assets::loaders"
         pub message: String,
         pub fields: serde_json::Value, // Store structured fields as a JSON Value
     }
     
     #[derive(Resource)]
     pub struct LogReloadHandle(pub reload::Handle<EnvFilter, Registry>);
     
     #[derive(Resource, Clone)]
     pub struct CentralLogStore(pub Arc<Mutex<VecDeque<ProcessedLog>>>);

  2. Create the Processing & Storage Layers: These will be new tracing::Layer implementations within the logging module.
     • DeduplicationLayer: A tracing::Layer that performs hash-based deduplication. It will store a HashMap<u64, Instant> to track the last time a log hash was seen. It will be configurable to only drop events if the same hash appeared within a specified Duration.
     • StoreLayer: A tracing::Layer that:
       a. Collects all metadata from a tracing::Event and its span.
       b. Constructs a ProcessedLog struct.
       c. Locks the CentralLogStore, pushes the new log, and enforces the maximum buffer size from LoggingConfig.
  3. Initialize the Service in whip_ui_example/src/main.rs:
     • Remove the existing LogPlugin.
     • Create the LoggingConfig instance: let config = LoggingConfig { central_store_capacity: 10_000, default_global_filter: "info,wgpu=error".to_string() };
     • Create the CentralLogStore using the config.
     • Create the reload_handle for the global EnvFilter.
     • Assemble the tracing pipeline: Global EnvFilter -> DeduplicationLayer -> StoreLayer.
     • Conditionally add a console output layer:

       let subscriber = Registry::default()
           .with(filter_layer)
           .with(DeduplicationLayer::new())
           .with(StoreLayer::new(log_store.clone()));
       
       // Only add the console formatter in debug builds
       #[cfg(debug_assertions)]
       let subscriber = subscriber.with(tracing_subscriber::fmt::layer());
       
       tracing::subscriber::set_global_default(subscriber).expect("Failed to set global default subscriber");

     • Insert config, LogReloadHandle(reload_handle), and log_store.clone() as Bevy resources.

• ✅ Testable Outcome: The application runs. Logs are processed and stored in the central buffer. In debug builds, logs also appear on the console. In release builds, they do not.

Phase 3: Cleanup of Obsolete Code

• Objective: To safely remove code and components that have been made redundant by the new architecture, using unit tests to prevent regressions.

• Actionable Steps:

  1. Identify Obsolete Components: DebugRingBuffer, StateChangeTracker, gui_framework/diagnostics.rs, and feature flags debug_logging and trace_logging.
  2. Plan for Removal:
     • Delete DebugRingBuffer: Its functionality is entirely replaced by the CentralLogStore. Delete the gui_framework/debug module and its plugin registration.
     • Refactor StateChangeTracker with a Testing Safety Net:
       a. Write Unit Tests: Create a new test module for the DeduplicationLayer. Write unit tests that specifically replicate the known inputs and expected outputs of the old StateChangeTracker's hashing and comparison logic.
       b. Implement and Verify: Implement the time-windowed deduplication logic in the DeduplicationLayer. Ensure all new unit tests pass, proving the new implementation is a correct replacement.
       c. Delete: Once verified, delete the StateChangeTracker struct and its associated systems.
     • Delete gui_framework/diagnostics.rs: The UiDiagnosticsPlugin and its related systems are fully superseded by the tracing-based approach planned in Phase 6. Delete the file and remove its registration.
     • Remove Feature Flags: Clean up Cargo.toml and the codebase by removing the debug_logging and trace_logging feature flags and any associated #[cfg(...)] attributes.

• ✅ Testable Outcome: The application compiles and runs correctly. The unit tests for the DeduplicationLayer pass, confirming the refactor was successful. The old components are gone.

Phase 4: Foundational CLI (Text-Mode UI)

• Objective: To create a fully functional, interactive CLI with a simple, text-based renderer, built on a formal Renderer trait to ensure future extensibility. This phase delivers the Minimum Viable Product.

• Actionable Steps:

  1. Define the TerminalRenderer Trait in cli/renderer.rs: This creates a formal contract for any UI renderer.

     // Represents all state needed for the renderer to draw a single frame.
     pub struct CliFrameState<'a> {
         pub logs: &'a [ProcessedLog],
         pub input_buffer: &'a str,
         pub status_message: &'a str,
         // ... other state like scroll position, filter text, etc.
     }
     
     // The formal contract for any CLI renderer.
     pub trait TerminalRenderer {
         fn draw(&mut self, state: &CliFrameState) -> anyhow::Result<()>;
         fn resize(&mut self, new_width: u16, new_height: u16);
     }

  2. Implement a BasicTerminalRenderer in cli/renderer.rs: This simple renderer will implement the TerminalRenderer trait. It will use crossterm directly for line-by-line output and will not depend on Taffy or asset loading.
  3. Define Commands in cli/commands.rs: Create the CliCommand enum and a CommandParser that handles the /command <subcommand> <value> syntax.
  4. Implement TerminalSession Guard in cli/session.rs: Create the struct with a Drop implementation to guarantee crossterm::terminal::disable_raw_mode() is always called, preventing a broken terminal on panic or exit.
  5. Implement CliPlugin and cli_main_loop in cli/plugin.rs:
     • The CliPlugin's build function will:
       a. Create a bounded crossbeam-channel (e.g., bounded(32)) for CliCommands.
       b. Insert a CliCommandReceiver resource into the Bevy App.
       c. Spawn the cli_main_loop thread, passing it the CentralLogStore and the command sender.
     • The cli_main_loop will:
       a. Instantiate the TerminalSession guard.
       b. Create an instance of the BasicTerminalRenderer.
       c. Loop:
       i. Handle user input and parse commands using the CommandParser. Send commands intended for Bevy over the channel.
       ii. Lock the CentralLogStore, clone log data, and immediately unlock.
       iii. Apply its local display filter to the cloned data.
       iv. Create a CliFrameState and pass it to renderer.draw().
  6. Implement handle_cli_commands_system: This Bevy system checks the CliCommandReceiver channel for commands (like ExitApp or changing the global log filter) and executes them.

• ✅ Testable Outcome: A text-only CLI appears immediately on launch. Logs are visible and scrollable. Commands like /level debug and /exit are functional. The application exits cleanly, restoring the terminal to a usable state.

Phase 5: Advanced UI Integration (ratatui + Taffy)

• Objective: To replace the basic text-mode renderer with a sophisticated renderer that uses Taffy for layout computation and ratatui for rendering TUI widgets, with robust error handling for asset loading.

• Actionable Steps:

  1. Implement the RatatuiTaffyRenderer in cli/renderer.rs:
     • This new renderer will implement the TerminalRenderer trait.
     • It will be responsible for interpreting a UiDefinition, using Taffy to compute a layout, and then using ratatui to draw the UI with crossterm.
  2. Update CliPlugin in cli/plugin.rs:
     • Modify the plugin to create a second bounded crossbeam-channel for sending the loaded layout from Bevy to the CLI thread. This channel will transport a Result<UiDefinition, String>.
     • The plugin will now use the AssetServer to request the loading of assets/cli/layout/main.json.
  3. Update assets/systems.rs (ui_asset_loaded_system):
     • Implement logic to detect when the CLI's layout asset has finished loading.
     • On success, send Ok(cloned_definition) over the layout channel.
     • On failure (e.g., asset not found), send Err("Failed to load layout asset: ...".to_string()).
  4. Modify cli_main_loop:
     • The loop will now start by performing a non-blocking check on the layout channel.
     • If it receives Ok(definition), it will replace its BasicTerminalRenderer with a new RatatuiTaffyRenderer initialized with the definition.
     • If it receives Err(message), it will display the error in its status bar and continue to use the BasicTerminalRenderer, ensuring the CLI remains functional even if the layout is broken.

• ✅ Testable Outcome: The CLI now renders using the JSON-defined layout, drawn by ratatui. If the layout fails to load, the CLI gracefully falls back to the basic text-mode UI and displays an error message.

Phase 6: Diagnostics Integration

• Objective: To integrate Bevy and Vulkan diagnostic information into the logging pipeline, demonstrating the system's extensibility.

• Actionable Steps:

  1. Create diagnostics/mod.rs and diagnostics/systems.rs:
  2. Define Diagnostic Emitters: Create functions that emit structured tracing::event!s with a specific target like "diagnostics::frame_time".

     // Example emitter
     tracing::event!(
         target: "diagnostics::frame_time",
         Level::INFO,
         fps = 1.0 / time.delta_seconds_f64(),
         delta_ms = time.delta_seconds_f32() * 1000.0
     );

  3. Create Bevy Systems: Create systems that gather data (e.g., from bevy_diagnostic::Diagnostics or bevy::time::Time) and call the emitters.
  4. Add Systems to App: Register these new diagnostic systems in the WhipUiPlugin.
  5. Update CLI: The CLI can now filter for these diagnostics using its local display filter (e.g., show only logs where target starts with "diagnostics"). No changes are needed to the core logging service.

• ✅ Testable Outcome: Frame rate and other diagnostic data appear as structured logs in the CLI when the appropriate display filter is set.

Phase 7: Final Validation and Documentation

• Objective: Verify the entire system's correctness, performance, and usability, and provide clear documentation for future developers.

• Actionable Steps:

  1. [ ] Functional Validation: Confirm all CLI commands work. Confirm the two-level filtering works as intended. Confirm the application exits cleanly and restores the terminal state. Confirm the unit tests for the DeduplicationLayer are in place and passing.
  2. [ ] Performance Validation: Manually verify that the application's main render loop remains responsive even when the CLI is actively processing and displaying a large number of logs.
  3. [ ] Documentation (LOGGING_GUIDE.md): Create a comprehensive guide documenting the hybrid push/pull architecture, the two levels of filtering, CLI usage, the command system, and the service-consumer model.

• ✅ Testable Outcome: All validation checklist items are checked. The LOGGING_GUIDE.md is complete and committed to the repository.

[MrScripty]

The previous commits were following an earlier iteration of the development proposal. I am shifting the direction towards a better architecture which means we will be making some changes that invalidate the previous commits.

[MrScripty]

Phase Plan Update - Splitting Phase 5

After implementing Phase 4, we discovered some limitations with the pure crossterm approach that affect
usability:

1. Double typing: Crossterm event handling causes duplicate key presses
2. App exit issue: CLI exits but main thread continues, leaving terminal locked
3. Screen flickering: Excessive redraws without proper state management

Proposed Phase Restructuring:

Phase 4 (Current - COMPLETED with fixes):

• ✅ Basic crossterm CLI with TerminalRenderer trait
• 🔧 Fix critical issues: double typing and app exit

Phase 5 (NEW - Ratatui Migration):

• Migrate existing CLI to ratatui for better input handling and rendering
• Keep same functionality but with professional TUI framework
• No Taffy integration yet - pure ratatui widgets

Phase 6 (Formerly Phase 5 - Advanced UI):

• Add Taffy layout integration to ratatui
• JSON-defined layouts and advanced rendering
• RatatuiTaffyRenderer implementation

This split allows us to:

• Make Phase 4 CLI actually usable with minimal fixes
• Provide smoother migration path from crossterm → ratatui → ratatui+taffy
• Address real usability issues discovered during implementation
• Maintain architectural progression while fixing immediate problems

Remaining phases shift accordingly (former Phase 6 becomes Phase 7, etc.)

Phase 4 Fixes:

• Fix crossterm double typing in event handling
• Fix main thread blocking on CLI exit
• Minimal flickering improvements