Conversation
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling. - Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth. - Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
- Updated `services.py` to allow service restart with an option to recreate containers, addressing WSL2 bind mount issues. - Added new chat configuration management functions in `system_controller.py` for loading, saving, and validating chat prompts. - Introduced `ChatSettings` component in the web UI for admin users to manage chat configurations easily. - Updated API service methods in `api.ts` to support chat configuration endpoints. - Integrated chat settings into the system management page for better accessibility.
…ion logging - Updated `start.sh` to improve shutdown handling by explicitly killing the backend process if running. - Modified `chat_service.py` to enhance logging for loading chat system prompts, providing clearer feedback on configuration usage. - Added a new `chat` field in `model_registry.py` for better chat service configuration management. - Updated vector store query parameters in `vector_stores.py` for improved clarity and functionality. - Enhanced the chat component in the web UI to conditionally auto-scroll based on message sending status.
…management - Introduced a new plugin architecture to allow for extensibility in the Chronicle application. - Added Home Assistant plugin for controlling devices via natural language commands triggered by wake words. - Implemented plugin configuration management endpoints in the API for loading, saving, and validating plugin settings. - Enhanced the web UI with a dedicated Plugins page for managing plugin configurations. - Updated Docker Compose files to include Tailscale integration for remote service access. - Refactored existing services to support plugin interactions during conversation and memory processing. - Improved error handling and logging for plugin initialization and execution processes.
- Updated .gitignore to include plugins.yml for security reasons. - Modified start.sh to allow passing additional arguments during service startup. - Refactored wizard.py to support new HF_TOKEN configuration prompts and improved handling of wake words in plugin settings. - Introduced a new setup_hf_token_if_needed function to streamline Hugging Face token management. - Enhanced the GitHub Actions workflow to create plugins.yml from a template, ensuring proper configuration setup. - Added detailed comments and documentation in the plugins.yml.template for better user guidance on Home Assistant integration.
…word processing - Added asynchronous Redis support in ClientManager for tracking client-user relationships. - Introduced `initialize_redis_for_client_manager` to set up Redis for cross-container mapping. - Updated `create_client_state` to use asynchronous tracking for client-user relationships. - Enhanced wake word processing in PluginRouter with normalization and command extraction. - Refactored DeepgramStreamingConsumer to utilize async Redis lookups for user ID retrieval. - Set TTL on Redis streams during client state cleanup for better resource management.
- Disabled the batch Deepgram worker in favor of the streaming worker to prevent race conditions. - Updated text normalization in wake word processing to replace punctuation with spaces, preserving word boundaries. - Enhanced regex pattern for wake word matching to allow optional punctuation and whitespace after the last part. - Improved logging in DeepgramStreamingConsumer for better visibility of message processing and error handling.
- Implemented retrieval of the original chat prompt before saving a custom prompt to ensure test isolation. - Added restoration of the original prompt after the test to prevent interference with subsequent tests. - Enhanced the test documentation for clarity on the purpose of these changes.
- Simplified test execution commands in CLAUDE.md and quickstart.md for better usability. - Added instructions for running tests from the project root and clarified the process for executing the complete Robot Framework test suite. - Introduced a new Docker service for the Deepgram streaming worker in docker-compose-test.yml to improve testing capabilities. - Updated system_admin_tests.robot to use a defined default prompt for restoration, enhancing test reliability and clarity.
- Updated `run-test.sh` and `run-robot-tests.sh` to improve cleanup processes, including handling permission issues with Docker. - Introduced a new function `mark_session_complete` in `session_controller.py` to ensure atomic updates for session completion status. - Refactored WebSocket and conversation job handling to utilize the new session completion function, enhancing reliability. - Updated `start-workers.sh` to enable the batch Deepgram worker alongside the streaming worker for improved transcription capabilities. - Enhanced test scripts to verify the status of Deepgram workers and ensure proper cleanup of test containers.
…ocess handling - Replaced the bash-based `start-workers.sh` script with a Python-based worker orchestrator for better process management and health monitoring. - Updated `docker-compose.yml` to configure the new orchestrator and adjust worker definitions, including the addition of audio persistence and stream workers. - Enhanced the Dockerfile to remove the old startup script and ensure the orchestrator is executable. - Introduced new modules for orchestrator configuration, health monitoring, process management, and worker registry to streamline worker lifecycle management. - Improved environment variable handling for worker configuration and health checks.
- Deleted the `run-test.sh` script, which was used for local test execution. - Updated Docker configurations to replace the `start-workers.sh` script with `worker_orchestrator.py` for improved worker management. - Enhanced health monitoring and process management in the orchestrator to ensure better reliability and logging. - Adjusted deployment configurations to reflect the new orchestrator setup.
- Introduced a new method `_handle_registration_loss` to manage RQ worker registration loss, replicating the behavior of the previous bash script. - Implemented a cooldown period to prevent frequent restarts during network issues. - Added logging for bulk restart actions and their outcomes to enhance monitoring and debugging capabilities. - Created a `_restart_all_rq_workers` method to facilitate the bulk restart of RQ workers, ensuring they re-register with Redis upon startup.
…tion - Introduced a new Test Event Plugin to log all plugin events to an SQLite database for integration testing. - Updated the plugin system to utilize event subscriptions instead of access levels, allowing for more flexible event handling. - Refactored the PluginRouter to dispatch events based on subscriptions, improving the event-driven architecture. - Enhanced Docker configurations to support development and testing environments with appropriate dependencies. - Added comprehensive integration tests to verify the functionality of the event dispatch system and plugin interactions. - Updated documentation and test configurations to reflect the new event-based plugin structure.
- Updated `docker-compose-test.yml` to include a test command for services, enabling a dedicated test mode. - Modified `start.sh` to support a `--test` flag, allowing the FastAPI backend to run with test-specific configurations. - Adjusted worker commands to utilize the `--group test` option in test mode for improved orchestration and management.
- Updated `run-robot-tests.sh` to enhance the verification of the Deepgram batch worker process, ensuring non-numeric characters are removed from the check. - Modified `plugin_tests.robot` to use a more explicit method for checking the length of subscriptions and added a skip condition for unavailable audio files. - Adjusted `plugin_event_tests.robot` to load the test audio file from a variable, improving test data management. - Refactored `plugin_keywords.robot` to utilize clearer length checks for subscriptions and event parts, enhancing readability and maintainability.
- Updated WebSocket endpoints to use a unified format with codec parameters (`/ws?codec=pcm` and `/ws?codec=opus`) for audio streaming, replacing the previous `/ws_pcm` and `/ws_omi` endpoints. - Enhanced documentation to reflect the new endpoint structure and clarify audio processing capabilities. - Removed deprecated audio cropping functionality and related configurations to streamline the audio processing workflow. - Updated various components and scripts to align with the new endpoint structure, ensuring consistent usage across the application.
- Updated `docker-compose-test.yml` to introduce low speech detection thresholds for testing, improving the accuracy of speech detection during tests. - Added new test-only API routes in `test_routes.py` for clearing and retrieving plugin events, ensuring a clean state between tests. - Refactored existing test scripts to utilize the new API endpoints for event management, enhancing test reliability and clarity. - Improved logging and error handling in various components to facilitate debugging during test execution. - Adjusted environment variable handling in test setup scripts to streamline configuration and improve flexibility.
…stence worker configuration - Introduced a comprehensive documentation file detailing the audio pipeline architecture, covering data flow, processing stages, and key components. - Enhanced the audio persistence worker setup by implementing multiple concurrent workers to improve audio processing efficiency. - Adjusted sleep intervals in the audio streaming persistence job for better responsiveness and event loop yielding. - Updated test script to run the full suite of integration tests from the specified directory, ensuring thorough testing coverage.
- Introduced `setup-test-containers.sh` for streamlined startup of test containers, including health checks and environment variable loading. - Added `teardown-test-containers.sh` for simplified container shutdown, with options to remove volumes. - Enhanced user feedback with color-coded messages for better visibility during test setup and teardown processes.
- Adjusted worker count expectations in the Worker Count Validation Test to reflect an increase from 7 to 9 workers, accounting for additional audio persistence workers. - Enhanced the WebSocket Disconnect Conversation End Reason Test by adding steps to maintain audio streaming during disconnection, ensuring accurate simulation of network dropout scenarios. - Improved comments for clarity and added critical notes regarding inactivity timeout handling.
… management - Replaced the legacy AudioFile model with AudioChunkDocument for storing audio data in MongoDB, optimizing storage and retrieval. - Introduced CleanupSettings dataclass for managing soft-deletion configurations, including auto-cleanup and retention days. - Added admin API routes for retrieving and saving cleanup settings, ensuring better control over data retention policies. - Updated audio processing workflows to utilize MongoDB chunks, removing dependencies on disk-based audio files. - Enhanced tests to validate the new audio chunk storage and cleanup functionalities, ensuring robust integration with existing systems.
…andling - Removed audio file path parameters from various functions, transitioning to audio data retrieval from MongoDB chunks. - Updated the `start_post_conversation_jobs` function to reflect changes in audio handling, ensuring jobs reconstruct audio from database chunks. - Enhanced the `transcribe_full_audio_job` and `recognise_speakers_job` to process audio directly from memory, eliminating the need for temporary files. - Improved error handling and logging for audio data retrieval, ensuring better feedback during processing. - Added a new utility function for converting PCM data to WAV format in memory, streamlining audio format handling.
- Updated the plugin setup script to run interactively, enabling plugins to prompt for user input during configuration. - Removed output capturing to facilitate real-time interaction and improved error messaging to include exit codes for better debugging.
- Introduced `setup_utils.py` containing functions for reading environment variables, prompting user input, and masking sensitive values. - Refactored existing code in `wizard.py` and `init.py` to utilize these shared utilities, improving code reuse and maintainability. - Updated documentation to include usage examples for the new utilities in plugin setup scripts, enhancing developer experience and clarity.
- Introduced a three-file separation for plugin configuration to improve security:
- `backends/advanced/.env` for secrets (gitignored)
- `config/plugins.yml` for orchestration with environment variable references
- `plugins/{plugin_id}/config.yml` for non-secret defaults
- Updated documentation to emphasize the importance of using `${ENV_VAR}` syntax for sensitive data and provided examples of correct usage.
- Enhanced the Email Summarizer plugin setup process to automatically update `config/plugins.yml` with environment variable references, ensuring secrets are not hardcoded.
- Added new fields to the User model for notification email management and improved error logging in user-related functions.
- Refactored audio chunk utilities to use a consistent method for fetching conversation metadata.
- Added a new parameter `transcript_version_id` to the `open_conversation_job` function to support streaming transcript versioning. - Enhanced error handling in `check_enrolled_speakers_job` and `recognise_speakers_job` to allow conversations to proceed even when the speaker service is unavailable, improving resilience. - Updated `send_to_adv.py` to support dynamic WebSocket and HTTP protocols based on environment settings, enhancing configuration flexibility. - Introduced a background task in `send_to_adv.py` to handle incoming messages from the backend, ensuring connection stability and logging interim results.
…nd error handling - Updated the controller to filter out invalid settings instead of raising an error for each unknown key, allowing for more flexible input. - Added a check to reject requests with no valid settings provided, enhancing robustness. - Adjusted logging to reflect the filtered settings being saved.
…duplication and tracking
…rity - Updated `mask_value` function to handle whitespace more effectively. - Enhanced `create_plugin` to remove existing directories when using the `--force` option. - Changed logging level from error to debug for existing admin user checks. - Improved client ID generation logging for clarity. - Removed unused fields from conversation creation. - Added HTML escaping in email templates to prevent XSS attacks. - Updated audio file download function to include user ID for better tracking. - Adjusted WebSocket connection settings to respect SSL verification based on environment variables.
- Removed `auto_generate_client` and `folder` parameters from audio upload functions to streamline the API. - Updated related function calls and documentation to reflect these changes, enhancing clarity and reducing complexity.
…nd security - Removed outdated migration instructions from `plugin-configuration.md` to streamline documentation. - Enhanced `README.md` to clearly outline the three-file separation for plugin configuration, emphasizing the roles of `.env`, `config.yml`, and `plugins.yml`. - Updated `setup.py` to reflect changes in orchestration settings, ensuring only relevant configurations are included in `config/plugins.yml`. - Improved security messaging to highlight the importance of not committing secrets to version control.
…t variable syntax for improved flexibility and security. This change standardizes the way API keys are referenced across different models and services. (SimpleOpenSoftware#273) Co-authored-by: roshan.john <roshanjohn1460@gmail.com>
- Replaced total job count with separate counters for successful and failed jobs during Redis queue cleanup. - Enhanced logging to provide detailed feedback on the number of jobs cleared and any failures encountered. - Improved error handling to ensure job counts are accurately reflected even when exceptions occur.
…ed container status check - Replaced 'docker logs' commands with 'docker compose -f docker-compose-test.yml logs' for consistency across workflows. - Added a check for running containers before saving logs to enhance debugging capabilities.
…amp adjustments - Added `audio_offset_seconds` to track cumulative audio duration for accurate timestamp adjustments across transcription sessions. - Updated `store_final_result` method to adjust word and segment timestamps based on cumulative audio offset. - Improved logging to reflect changes in audio offset after storing results. - Modified Makefile and documentation to clarify test execution options, including new tags for slow and SDK tests, enhancing test organization and execution clarity.
…n tests - Set `COMPOSE_PROJECT_NAME` for test containers to ensure consistent naming. - Consolidated error messages in the `websocket_transcription_e2e_test.robot` file for clarity, improving readability and debugging.
- Added timeout handling for WebSocket closure in `AudioStreamClient` to prevent hanging and ensure clean disconnection. - Updated integration tests to log the total chunks sent when closing audio streams, improving clarity on resource management during test teardown.
- Updated job status checks across various modules to use "started" and "finished" instead of "processing" and "completed" for consistency with RQ's naming conventions. - Adjusted related logging and response messages to reflect the new status terminology. - Simplified Docker Compose project name handling in test scripts to avoid conflicts and improve clarity in test environment setup.
- Increased `SPEECH_INACTIVITY_THRESHOLD_SECONDS` to 20 seconds in `docker-compose-test.yml` for better audio duration handling during tests. - Refactored session handling in `session_controller.py` to clarify client ID usage. - Updated `conversation_utils.py` to track speech activity using audio timestamps, enhancing accuracy in inactivity detection. - Simplified test scripts by removing unnecessary `COMPOSE_PROJECT_NAME` references, aligning with the new project naming convention. - Adjusted integration tests to reflect changes in inactivity timeout and ensure proper handling of audio timestamps.
- Updated `worker_orchestrator.py` to use `logger.exception` for improved error logging. - Changed default MongoDB database name from "friend-lite" to "chronicle" in multiple files for consistency. - Added a new method `close_stream_without_stop` in `audio_stream_client.py` to handle abrupt WebSocket disconnections. - Enhanced audio validation in `audio_utils.py` to support automatic resampling of audio data if sample rates do not match. - Improved logging in various modules to provide clearer insights during audio processing and event dispatching.
- Updated `run_compose_command` to support separate build commands for services, including profile management for backend and speaker-recognition services. - Improved error handling and output streaming during Docker command execution. - Added `ensure_docker_network` function to verify and create the required Docker network before starting services. - Refactored configuration files to utilize `oc.env` for environment variable management, ensuring better compatibility and flexibility across different environments.
- Added support for the CONFIG_FILE environment variable to allow specifying custom configuration files for testing. - Implemented logic to handle both absolute paths and relative filenames for the configuration file, improving flexibility in configuration management.
- Replaced CONFIG_FILE with TEST_CONFIG_FILE in both run-no-api-tests.sh and run-robot-tests.sh to standardize configuration file usage. - Updated paths to point to mock and deepgram-openai configuration files inside the container, improving clarity and consistency in test setups.
- Updated `upload_and_process_audio_files` to return appropriate HTTP status codes based on upload results: 400 for all failures, 207 for partial successes, and 200 for complete success. - Enhanced error messages in the audio upload tests to provide clearer feedback on upload failures, including specific error details for better debugging. - Adjusted test scripts to ensure consistent handling of conversation IDs in job metadata, improving validation checks for job creation.
Contributor
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the
✨ Finishing touches🧪 Generate unit tests (beta)
Tip 🧪 Unit Test Generation v2 is now available!We have significantly improved our unit test generation capabilities. To enable: Add this to your reviews:
finishing_touches:
unit_tests:
enabled: trueTry it out by using the Have feedback? Share your thoughts on our Discord thread! Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.