Open
Conversation
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling. - Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth. - Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
- Updated `services.py` to allow service restart with an option to recreate containers, addressing WSL2 bind mount issues. - Added new chat configuration management functions in `system_controller.py` for loading, saving, and validating chat prompts. - Introduced `ChatSettings` component in the web UI for admin users to manage chat configurations easily. - Updated API service methods in `api.ts` to support chat configuration endpoints. - Integrated chat settings into the system management page for better accessibility.
…ion logging - Updated `start.sh` to improve shutdown handling by explicitly killing the backend process if running. - Modified `chat_service.py` to enhance logging for loading chat system prompts, providing clearer feedback on configuration usage. - Added a new `chat` field in `model_registry.py` for better chat service configuration management. - Updated vector store query parameters in `vector_stores.py` for improved clarity and functionality. - Enhanced the chat component in the web UI to conditionally auto-scroll based on message sending status.
…management - Introduced a new plugin architecture to allow for extensibility in the Chronicle application. - Added Home Assistant plugin for controlling devices via natural language commands triggered by wake words. - Implemented plugin configuration management endpoints in the API for loading, saving, and validating plugin settings. - Enhanced the web UI with a dedicated Plugins page for managing plugin configurations. - Updated Docker Compose files to include Tailscale integration for remote service access. - Refactored existing services to support plugin interactions during conversation and memory processing. - Improved error handling and logging for plugin initialization and execution processes.
- Updated .gitignore to include plugins.yml for security reasons. - Modified start.sh to allow passing additional arguments during service startup. - Refactored wizard.py to support new HF_TOKEN configuration prompts and improved handling of wake words in plugin settings. - Introduced a new setup_hf_token_if_needed function to streamline Hugging Face token management. - Enhanced the GitHub Actions workflow to create plugins.yml from a template, ensuring proper configuration setup. - Added detailed comments and documentation in the plugins.yml.template for better user guidance on Home Assistant integration.
…word processing - Added asynchronous Redis support in ClientManager for tracking client-user relationships. - Introduced `initialize_redis_for_client_manager` to set up Redis for cross-container mapping. - Updated `create_client_state` to use asynchronous tracking for client-user relationships. - Enhanced wake word processing in PluginRouter with normalization and command extraction. - Refactored DeepgramStreamingConsumer to utilize async Redis lookups for user ID retrieval. - Set TTL on Redis streams during client state cleanup for better resource management.
- Disabled the batch Deepgram worker in favor of the streaming worker to prevent race conditions. - Updated text normalization in wake word processing to replace punctuation with spaces, preserving word boundaries. - Enhanced regex pattern for wake word matching to allow optional punctuation and whitespace after the last part. - Improved logging in DeepgramStreamingConsumer for better visibility of message processing and error handling.
- Implemented retrieval of the original chat prompt before saving a custom prompt to ensure test isolation. - Added restoration of the original prompt after the test to prevent interference with subsequent tests. - Enhanced the test documentation for clarity on the purpose of these changes.
- Simplified test execution commands in CLAUDE.md and quickstart.md for better usability. - Added instructions for running tests from the project root and clarified the process for executing the complete Robot Framework test suite. - Introduced a new Docker service for the Deepgram streaming worker in docker-compose-test.yml to improve testing capabilities. - Updated system_admin_tests.robot to use a defined default prompt for restoration, enhancing test reliability and clarity.
- Updated `run-test.sh` and `run-robot-tests.sh` to improve cleanup processes, including handling permission issues with Docker. - Introduced a new function `mark_session_complete` in `session_controller.py` to ensure atomic updates for session completion status. - Refactored WebSocket and conversation job handling to utilize the new session completion function, enhancing reliability. - Updated `start-workers.sh` to enable the batch Deepgram worker alongside the streaming worker for improved transcription capabilities. - Enhanced test scripts to verify the status of Deepgram workers and ensure proper cleanup of test containers.
…ocess handling - Replaced the bash-based `start-workers.sh` script with a Python-based worker orchestrator for better process management and health monitoring. - Updated `docker-compose.yml` to configure the new orchestrator and adjust worker definitions, including the addition of audio persistence and stream workers. - Enhanced the Dockerfile to remove the old startup script and ensure the orchestrator is executable. - Introduced new modules for orchestrator configuration, health monitoring, process management, and worker registry to streamline worker lifecycle management. - Improved environment variable handling for worker configuration and health checks.
- Deleted the `run-test.sh` script, which was used for local test execution. - Updated Docker configurations to replace the `start-workers.sh` script with `worker_orchestrator.py` for improved worker management. - Enhanced health monitoring and process management in the orchestrator to ensure better reliability and logging. - Adjusted deployment configurations to reflect the new orchestrator setup.
- Introduced a new method `_handle_registration_loss` to manage RQ worker registration loss, replicating the behavior of the previous bash script. - Implemented a cooldown period to prevent frequent restarts during network issues. - Added logging for bulk restart actions and their outcomes to enhance monitoring and debugging capabilities. - Created a `_restart_all_rq_workers` method to facilitate the bulk restart of RQ workers, ensuring they re-register with Redis upon startup.
…tion - Introduced a new Test Event Plugin to log all plugin events to an SQLite database for integration testing. - Updated the plugin system to utilize event subscriptions instead of access levels, allowing for more flexible event handling. - Refactored the PluginRouter to dispatch events based on subscriptions, improving the event-driven architecture. - Enhanced Docker configurations to support development and testing environments with appropriate dependencies. - Added comprehensive integration tests to verify the functionality of the event dispatch system and plugin interactions. - Updated documentation and test configurations to reflect the new event-based plugin structure.
- Updated `docker-compose-test.yml` to include a test command for services, enabling a dedicated test mode. - Modified `start.sh` to support a `--test` flag, allowing the FastAPI backend to run with test-specific configurations. - Adjusted worker commands to utilize the `--group test` option in test mode for improved orchestration and management.
- Updated `run-robot-tests.sh` to enhance the verification of the Deepgram batch worker process, ensuring non-numeric characters are removed from the check. - Modified `plugin_tests.robot` to use a more explicit method for checking the length of subscriptions and added a skip condition for unavailable audio files. - Adjusted `plugin_event_tests.robot` to load the test audio file from a variable, improving test data management. - Refactored `plugin_keywords.robot` to utilize clearer length checks for subscriptions and event parts, enhancing readability and maintainability.
- Updated WebSocket endpoints to use a unified format with codec parameters (`/ws?codec=pcm` and `/ws?codec=opus`) for audio streaming, replacing the previous `/ws_pcm` and `/ws_omi` endpoints. - Enhanced documentation to reflect the new endpoint structure and clarify audio processing capabilities. - Removed deprecated audio cropping functionality and related configurations to streamline the audio processing workflow. - Updated various components and scripts to align with the new endpoint structure, ensuring consistent usage across the application.
- Updated `docker-compose-test.yml` to introduce low speech detection thresholds for testing, improving the accuracy of speech detection during tests. - Added new test-only API routes in `test_routes.py` for clearing and retrieving plugin events, ensuring a clean state between tests. - Refactored existing test scripts to utilize the new API endpoints for event management, enhancing test reliability and clarity. - Improved logging and error handling in various components to facilitate debugging during test execution. - Adjusted environment variable handling in test setup scripts to streamline configuration and improve flexibility.
…stence worker configuration - Introduced a comprehensive documentation file detailing the audio pipeline architecture, covering data flow, processing stages, and key components. - Enhanced the audio persistence worker setup by implementing multiple concurrent workers to improve audio processing efficiency. - Adjusted sleep intervals in the audio streaming persistence job for better responsiveness and event loop yielding. - Updated test script to run the full suite of integration tests from the specified directory, ensuring thorough testing coverage.
- Introduced `setup-test-containers.sh` for streamlined startup of test containers, including health checks and environment variable loading. - Added `teardown-test-containers.sh` for simplified container shutdown, with options to remove volumes. - Enhanced user feedback with color-coded messages for better visibility during test setup and teardown processes.
- Adjusted worker count expectations in the Worker Count Validation Test to reflect an increase from 7 to 9 workers, accounting for additional audio persistence workers. - Enhanced the WebSocket Disconnect Conversation End Reason Test by adding steps to maintain audio streaming during disconnection, ensuring accurate simulation of network dropout scenarios. - Improved comments for clarity and added critical notes regarding inactivity timeout handling.
… management - Replaced the legacy AudioFile model with AudioChunkDocument for storing audio data in MongoDB, optimizing storage and retrieval. - Introduced CleanupSettings dataclass for managing soft-deletion configurations, including auto-cleanup and retention days. - Added admin API routes for retrieving and saving cleanup settings, ensuring better control over data retention policies. - Updated audio processing workflows to utilize MongoDB chunks, removing dependencies on disk-based audio files. - Enhanced tests to validate the new audio chunk storage and cleanup functionalities, ensuring robust integration with existing systems.
…andling - Removed audio file path parameters from various functions, transitioning to audio data retrieval from MongoDB chunks. - Updated the `start_post_conversation_jobs` function to reflect changes in audio handling, ensuring jobs reconstruct audio from database chunks. - Enhanced the `transcribe_full_audio_job` and `recognise_speakers_job` to process audio directly from memory, eliminating the need for temporary files. - Improved error handling and logging for audio data retrieval, ensuring better feedback during processing. - Added a new utility function for converting PCM data to WAV format in memory, streamlining audio format handling.
…ed container status check - Replaced 'docker logs' commands with 'docker compose -f docker-compose-test.yml logs' for consistency across workflows. - Added a check for running containers before saving logs to enhance debugging capabilities.
…amp adjustments - Added `audio_offset_seconds` to track cumulative audio duration for accurate timestamp adjustments across transcription sessions. - Updated `store_final_result` method to adjust word and segment timestamps based on cumulative audio offset. - Improved logging to reflect changes in audio offset after storing results. - Modified Makefile and documentation to clarify test execution options, including new tags for slow and SDK tests, enhancing test organization and execution clarity.
…n tests - Set `COMPOSE_PROJECT_NAME` for test containers to ensure consistent naming. - Consolidated error messages in the `websocket_transcription_e2e_test.robot` file for clarity, improving readability and debugging.
- Added timeout handling for WebSocket closure in `AudioStreamClient` to prevent hanging and ensure clean disconnection. - Updated integration tests to log the total chunks sent when closing audio streams, improving clarity on resource management during test teardown.
- Updated job status checks across various modules to use "started" and "finished" instead of "processing" and "completed" for consistency with RQ's naming conventions. - Adjusted related logging and response messages to reflect the new status terminology. - Simplified Docker Compose project name handling in test scripts to avoid conflicts and improve clarity in test environment setup.
- Increased `SPEECH_INACTIVITY_THRESHOLD_SECONDS` to 20 seconds in `docker-compose-test.yml` for better audio duration handling during tests. - Refactored session handling in `session_controller.py` to clarify client ID usage. - Updated `conversation_utils.py` to track speech activity using audio timestamps, enhancing accuracy in inactivity detection. - Simplified test scripts by removing unnecessary `COMPOSE_PROJECT_NAME` references, aligning with the new project naming convention. - Adjusted integration tests to reflect changes in inactivity timeout and ensure proper handling of audio timestamps.
- Updated `worker_orchestrator.py` to use `logger.exception` for improved error logging. - Changed default MongoDB database name from "friend-lite" to "chronicle" in multiple files for consistency. - Added a new method `close_stream_without_stop` in `audio_stream_client.py` to handle abrupt WebSocket disconnections. - Enhanced audio validation in `audio_utils.py` to support automatic resampling of audio data if sample rates do not match. - Improved logging in various modules to provide clearer insights during audio processing and event dispatching.
- Updated `run_compose_command` to support separate build commands for services, including profile management for backend and speaker-recognition services. - Improved error handling and output streaming during Docker command execution. - Added `ensure_docker_network` function to verify and create the required Docker network before starting services. - Refactored configuration files to utilize `oc.env` for environment variable management, ensuring better compatibility and flexibility across different environments.
- Added support for the CONFIG_FILE environment variable to allow specifying custom configuration files for testing. - Implemented logic to handle both absolute paths and relative filenames for the configuration file, improving flexibility in configuration management.
- Replaced CONFIG_FILE with TEST_CONFIG_FILE in both run-no-api-tests.sh and run-robot-tests.sh to standardize configuration file usage. - Updated paths to point to mock and deepgram-openai configuration files inside the container, improving clarity and consistency in test setups.
- Updated `upload_and_process_audio_files` to return appropriate HTTP status codes based on upload results: 400 for all failures, 207 for partial successes, and 200 for complete success. - Enhanced error messages in the audio upload tests to provide clearer feedback on upload failures, including specific error details for better debugging. - Adjusted test scripts to ensure consistent handling of conversation IDs in job metadata, improving validation checks for job creation.
…anagement - Updated `upload_and_process_audio_files` to check for transcription provider availability before enqueueing jobs, enhancing error handling and logging. - Modified `start_post_conversation_jobs` to conditionally enqueue memory extraction jobs based on configuration, improving flexibility in job management. - Enhanced event dispatch job dependencies to only include jobs that were actually enqueued, ensuring accurate job tracking. - Added `is_transcription_available` function to check transcription provider status, improving modularity and clarity in the transcription workflow.
- Updated integration tests to filter plugin events by conversation ID, ensuring accurate event tracking and reducing noise from fixture events. - Improved error messages in event verification to include conversation ID context, enhancing clarity during test failures. - Refactored audio upload handling to check for transcription job creation, allowing for more robust conversation polling and error reporting. - Added new keyword to verify conversation end reasons, improving test coverage for conversation state validation.
- Added mock speaker recognition client to facilitate testing without resource-intensive dependencies. - Updated Docker Compose configurations to include mock speaker client for test environments. - Refactored audio segment reconstruction to ensure precise clipping based on time boundaries. - Improved error handling in transcription jobs and speaker recognition workflows to enhance robustness. - Adjusted integration tests to utilize real-time pacing for audio chunk streaming, improving test accuracy.
- Introduced logging for audio chunk requests to improve traceability. - Replaced manual audio chunk processing with a dedicated `reconstruct_audio_segment` function for better clarity and efficiency. - Improved error handling during audio reconstruction to provide more informative responses in case of failures. - Cleaned up imports and removed redundant code related to audio chunk calculations.
- Replaced direct import of mock client with a structured import from the new testing module. - Introduced a dedicated `mock_speaker_client.py` to provide a mock implementation for speaker recognition, facilitating testing without heavy dependencies. - Added an `__init__.py` file in the testing directory to organize testing utilities and mocks.
…ve transcript handling - Added a new `words` field to the `Conversation` model for storing word-level timestamps. - Updated methods to handle word data during transcript version creation, ensuring compatibility with speaker recognition. - Refactored conversation job processing to utilize the new word structure, improving data integrity and access. - Enhanced speaker recognition job to read words from the new standardized location, ensuring backward compatibility with legacy data.
- Added a new endpoint to reprocess speaker identification for existing transcripts, creating a new version with re-identified speakers. - Introduced a method to calculate proportional timeouts based on audio duration, improving handling of varying audio lengths. - Updated the speaker recognition client to utilize calculated timeouts during service calls, enhancing responsiveness. - Refactored conversation and memory controllers to support the new speaker reprocessing functionality, ensuring user access control and job chaining for memory updates. - Removed unfiltered memory retrieval endpoint to streamline memory management and focus on user-specific data access.
…ng/export workflow
…nto feat/swipe-anomaly
- Updated the `maybe_anomaly` field in the `Conversation` model to include 'rejected' as a possible status. - Implemented logic to mark transcripts as handled in the database when rejected, preventing them from reappearing in the events queue. - Added a new environment variable to enable or disable the user loop modal in the frontend. - Refactored the `UserLoopModal` component to fetch events from the backend and conditionally display based on the new environment variable. - Improved error handling in event fetching to ensure robustness. - Updated tests to verify the correct handling of rejected transcripts and their impact on the events queue.
- Updated user loop integration tests to improve structure and clarity. - Added new keywords for marking test conversations as deleted in the helper functions. - Enhanced test cases to ensure proper handling of rejected and accepted swipe flows. - Improved documentation for test cases and keywords to reflect recent changes.
Contributor
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the
✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reopens #275