Merged
Conversation
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling. - Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth. - Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
- Updated `services.py` to allow service restart with an option to recreate containers, addressing WSL2 bind mount issues. - Added new chat configuration management functions in `system_controller.py` for loading, saving, and validating chat prompts. - Introduced `ChatSettings` component in the web UI for admin users to manage chat configurations easily. - Updated API service methods in `api.ts` to support chat configuration endpoints. - Integrated chat settings into the system management page for better accessibility.
Contributor
📝 WalkthroughWalkthroughAdds configurable chat system-prompt support end-to-end: backend registry/config fields, controller APIs for get/validate/save with hot-reload, routes, frontend ChatSettings UI and API clients, tests, and a service restart CLI option to support recreate restarts. Changes
Sequence Diagram(s)sequenceDiagram
actor Admin
participant UI as ChatSettings (Browser)
participant API as System Routes
participant Controller as System Controller
participant Registry as Models Registry
participant FS as Config File
Admin->>UI: Edit prompt YAML
UI->>API: POST /admin/chat/config/validate (text)
API->>Controller: validate_chat_config_yaml(prompt_text)
Controller-->>API: {valid: bool, errors?}
API-->>UI: Validation result
Admin->>UI: Click Save
UI->>API: POST /admin/chat/config (text)
API->>Controller: save_chat_config_yaml(prompt_text)
Controller->>FS: backup current config
Controller->>FS: write updated config (chat.system_prompt)
Controller->>Registry: load_models_config(force_reload=true)
Registry->>FS: read new config
Registry-->>Controller: updated registry (includes chat.system_prompt)
Controller-->>API: {success:true}
API-->>UI: Save result
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
Comment |
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Actions performedReview triggered.
|
Collaborator
Author
|
Still validating with a test |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 9
🧹 Nitpick comments (1)
services.py (1)
281-284: Consider warning about temporary downtime in recreate mode.The messaging helpfully explains which mode is active. However, when
recreate=True, services will experience temporary downtime during the down→up cycle. Consider adding a warning to manage user expectations.💡 Enhanced messaging example
if recreate: - console.print("[dim]Using down + up to recreate containers (fixes WSL2 bind mount issues)[/dim]\n") + console.print("[yellow]⚠️ Recreate mode: services will be temporarily unavailable during restart[/yellow]") + console.print("[dim]Using down + up to recreate containers (fixes WSL2 bind mount issues)[/dim]\n") else: console.print("[dim]Quick restart (use --recreate to fix bind mount issues)[/dim]\n")
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
backends/advanced/src/advanced_omi_backend/chat_service.pybackends/advanced/src/advanced_omi_backend/controllers/system_controller.pybackends/advanced/src/advanced_omi_backend/routers/modules/system_routes.pybackends/advanced/webui/src/components/ChatSettings.tsxbackends/advanced/webui/src/pages/System.tsxbackends/advanced/webui/src/services/api.tsservices.pytests/endpoints/system_admin_tests.robot
🧰 Additional context used
📓 Path-based instructions (3)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
TypeScript code should follow standard React Native conventions.
Files:
backends/advanced/webui/src/services/api.tsbackends/advanced/webui/src/components/ChatSettings.tsxbackends/advanced/webui/src/pages/System.tsx
**/*.robot
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.robot: When writing or modifying Robot Framework tests, read @tests/TESTING_GUIDELINES.md for comprehensive testing patterns and standards before implementation.
Robot Framework tests: Check @tests/tags.md for approved tags. ONLY use the 11 approved tags (tab-separated). Do not create custom tags.
Robot Framework tests: Before writing ANY test code, scan relevant resource files for existing keywords. NEVER write code that duplicates existing keywords.
Robot Framework tests: Write assertions directly in tests using inline verifications, not abstracted to keywords. Only create keywords for reusable setup/action operations.
Robot Framework tests: Use descriptive keyword and test names that explain business purpose, not technical implementation.
Files:
tests/endpoints/system_admin_tests.robot
**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.py: In Python backend, useuv run python3for running Python commands instead ofpython3directly, as the project usesuvpackage manager.
Python code formatting: Use Black formatter with 100-character line length and isort for import organization.
Python import guidelines: ALL imports must be at the top of the file after the docstring. NEVER import modules in the middle of functions or files. Group imports into standard library, third-party, and local imports. Use lazy imports sparingly and only when absolutely necessary for circular import issues.
Python error handling: Always raise errors, never silently ignore. Use explicit error handling with proper exceptions rather than silent failures. Do not add defensivehasattr()checks—research and understand input/response structures or class structure instead.
Files:
backends/advanced/src/advanced_omi_backend/chat_service.pybackends/advanced/src/advanced_omi_backend/controllers/system_controller.pybackends/advanced/src/advanced_omi_backend/routers/modules/system_routes.pyservices.py
🧠 Learnings (1)
📚 Learning: 2025-12-08T23:52:34.959Z
Learnt from: AnkushMalaker
Repo: chronicler-ai/chronicle PR: 178
File: backends/advanced/src/advanced_omi_backend/services/memory/providers/mycelia.py:218-223
Timestamp: 2025-12-08T23:52:34.959Z
Learning: In Python code (chronicle project), prefer logging.exception() inside except blocks to automatically log the full stack trace. When re-raising exceptions, always chain with 'raise ... from e' to preserve the original context; use 'raise ... from None' only if you explicitly want to suppress the context. This improves debuggability across Python files.
Applied to files:
backends/advanced/src/advanced_omi_backend/chat_service.pybackends/advanced/src/advanced_omi_backend/controllers/system_controller.pybackends/advanced/src/advanced_omi_backend/routers/modules/system_routes.pyservices.py
🧬 Code graph analysis (5)
backends/advanced/webui/src/components/ChatSettings.tsx (2)
backends/advanced/webui/src/contexts/AuthContext.tsx (1)
useAuth(115-121)backends/advanced/webui/src/services/api.ts (1)
systemApi(150-186)
backends/advanced/src/advanced_omi_backend/chat_service.py (1)
backends/advanced/src/advanced_omi_backend/model_registry.py (1)
get_models_registry(344-359)
backends/advanced/src/advanced_omi_backend/controllers/system_controller.py (1)
backends/advanced/src/advanced_omi_backend/model_registry.py (2)
_find_config_path(248-282)load_models_config(285-341)
backends/advanced/src/advanced_omi_backend/routers/modules/system_routes.py (1)
backends/advanced/src/advanced_omi_backend/controllers/system_controller.py (3)
get_chat_config_yaml(462-487)save_chat_config_yaml(490-537)validate_chat_config_yaml(540-557)
backends/advanced/webui/src/pages/System.tsx (1)
backends/advanced/webui/src/components/ChatSettings.tsx (1)
ChatSettings(10-195)
🪛 Ruff (0.14.10)
backends/advanced/src/advanced_omi_backend/chat_service.py
161-161: Do not catch blind exception: Exception
(BLE001)
backends/advanced/src/advanced_omi_backend/controllers/system_controller.py
483-483: Consider moving this statement to an else block
(TRY300)
486-486: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
497-497: Abstract raise to an inner function
(TRY301)
497-497: Avoid specifying long messages outside the exception class
(TRY003)
501-501: Abstract raise to an inner function
(TRY301)
501-501: Avoid specifying long messages outside the exception class
(TRY003)
503-503: Abstract raise to an inner function
(TRY301)
503-503: Avoid specifying long messages outside the exception class
(TRY003)
533-533: Consider moving this statement to an else block
(TRY300)
536-536: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
553-553: Consider moving this statement to an else block
(TRY300)
555-555: Do not catch blind exception: Exception
(BLE001)
556-556: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
557-557: Use explicit conversion flag
Replace with conversion flag
(RUF010)
backends/advanced/src/advanced_omi_backend/routers/modules/system_routes.py
135-135: Unused function argument: current_user
(ARG001)
135-135: Do not perform function call Depends in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
140-140: Do not catch blind exception: Exception
(BLE001)
141-141: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
142-142: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
148-148: Unused function argument: current_user
(ARG001)
148-148: Do not perform function call Depends in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
155-155: Consider moving this statement to an else block
(TRY300)
157-157: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
158-158: Do not catch blind exception: Exception
(BLE001)
159-159: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
160-160: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
166-166: Unused function argument: current_user
(ARG001)
166-166: Do not perform function call Depends in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
173-173: Consider moving this statement to an else block
(TRY300)
174-174: Do not catch blind exception: Exception
(BLE001)
175-175: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
176-176: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: robot-tests
🔇 Additional comments (11)
services.py (2)
277-277: LGTM: Backward-compatible signature.The
recreate=Falsedefault preserves existing behavior while cleanly exposing the new functionality.
365-366: LGTM: Clean CLI integration.The
--recreateflag is properly wired through the argument parser to therestart_servicesfunction, with clear help text explaining its purpose.Also applies to: 430-430
backends/advanced/src/advanced_omi_backend/chat_service.py (2)
422-423: LGTM! Dynamic system prompt retrieval implemented correctly.The integration of
_get_system_prompt()allows hot-reloading of chat configuration without service restart, which aligns well with the admin UI requirements.
146-171: The proposed refactor is incomplete and would break the code.The
hasattr(reg, 'config')check at line 155 is necessary because theAppModelsclass does not have aconfigfield. TheAppModelsclass only defines:defaults,models,memory, andspeaker_recognition. Removing thehasattr()check and usingreg.configdirectly would immediately raiseAttributeErrorwithout fixing the underlying issue.Additionally, the
system_controller.pyaccesses chat config differently—it readsconfig.ymldirectly viayaml.safe_load()and accesses the raw config dict. The code inchat_service.pyappears to be an incomplete refactor that expects aconfigattribute onAppModelsthat was never added.To address this properly:
- Add
configfield toAppModelsto store the raw configuration dict, OR- Refactor
chat_service.pyto readconfig.ymldirectly likesystem_controller.pydoes instead of relying onAppModels.config.The exception handling should be updated after the root cause is resolved. Once a proper config access pattern is established, narrow the exception handling to specific types (e.g.,
KeyErrorfor missing config sections,TypeErrorfor unexpected data structures).backends/advanced/webui/src/pages/System.tsx (2)
6-6: LGTM! Clean integration of ChatSettings component.The import follows the existing pattern for other system settings components.
751-754: LGTM! ChatSettings section properly placed.The full-width section placement is consistent with the MemorySettings section below (lines 756-759) and provides appropriate admin-only chat configuration UI.
tests/endpoints/system_admin_tests.robot (2)
222-222: Admin endpoint list correctly updated.The
/api/admin/chat/configendpoint is properly included in the non-admin access test to ensure authorization is enforced.
152-206: Comprehensive chat configuration test coverage.The three test cases cover:
- GET endpoint with plain text response validation
- Validation logic with both valid and invalid inputs
- Save and retrieve round-trip verification
The tests follow Robot Framework conventions, use approved tags (
infraandpermissions), and are well-documented.backends/advanced/webui/src/services/api.ts (1)
172-181: LGTM! Chat configuration API methods follow established patterns.The three methods mirror the memory configuration API structure above (lines 161-170) with consistent use of
Content-Type: text/plainfor YAML/text payloads. Endpoint paths match the backend routes.backends/advanced/webui/src/components/ChatSettings.tsx (1)
10-99: Well-structured component with clear UX patterns.The component follows React best practices:
- Proper state management for loading/validation/saving states
- Clear error and success messaging with auto-clear timeouts
- Admin-only rendering guard
- Disabled states during operations
backends/advanced/src/advanced_omi_backend/controllers/system_controller.py (1)
462-557: Well-structured chat configuration implementation.The three functions provide a clean API for managing chat configuration with appropriate validation, backup creation, and hot-reload support. The implementation follows the existing patterns in the controller (similar to memory config functions) and integrates well with the model registry system.
Comment on lines
+485
to
+487
| except Exception as e: | ||
| logger.error(f"Error loading chat config: {e}") | ||
| raise |
Contributor
There was a problem hiding this comment.
🛠️ Refactor suggestion | 🟠 Major
Improve exception handling to follow project patterns.
The exception handler should use logging.exception() instead of logger.error() to capture the full stack trace, and should re-raise with from e to preserve exception context.
Based on learnings, this improves debuggability.
🔎 Proposed fix
except Exception as e:
- logger.error(f"Error loading chat config: {e}")
- raise
+ logger.exception("Error loading chat config")
+ raise e📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| except Exception as e: | |
| logger.error(f"Error loading chat config: {e}") | |
| raise | |
| except Exception as e: | |
| logger.exception("Error loading chat config") | |
| raise |
🧰 Tools
🪛 Ruff (0.14.10)
486-486: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
🤖 Prompt for AI Agents
In backends/advanced/src/advanced_omi_backend/controllers/system_controller.py
around lines 485 to 487, the except block currently calls logger.error and
re-raises without preserving context; replace logger.error(...) with
logger.exception(...) to log the full stack trace, and re-raise the exception
using "raise ... from e" (e.g., raise <same-exception> from e) so the original
exception context is preserved.
Comment on lines
+535
to
+537
| except Exception as e: | ||
| logger.error(f"Error saving chat config: {e}") | ||
| raise |
Contributor
There was a problem hiding this comment.
🛠️ Refactor suggestion | 🟠 Major
Improve exception handling to follow project patterns.
The exception handler should use logging.exception() instead of logger.error() to capture the full stack trace, and should re-raise with from e to preserve exception context.
Based on learnings, this improves debuggability.
🔎 Proposed fix
except Exception as e:
- logger.error(f"Error saving chat config: {e}")
- raise
+ logger.exception("Error saving chat config")
+ raise e🧰 Tools
🪛 Ruff (0.14.10)
536-536: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
🤖 Prompt for AI Agents
In backends/advanced/src/advanced_omi_backend/controllers/system_controller.py
around lines 535 to 537, the except block uses logger.error and a plain raise;
replace logger.error(...) with logger.exception(...) to record the full stack
trace, and re-raise the caught exception using "raise ... from e" (i.e., raise
the same exception with "from e") to preserve exception chaining and context.
Comment on lines
+555
to
+557
| except Exception as e: | ||
| logger.error(f"Error validating chat config: {e}") | ||
| return {"valid": False, "error": f"Validation error: {str(e)}"} |
Contributor
There was a problem hiding this comment.
🛠️ Refactor suggestion | 🟠 Major
Use logging.exception() for better error diagnostics.
While the catch-all exception handling is appropriate here (since this is a validation function that returns error details), the logging should use logging.exception() instead of logger.error() to capture the full stack trace.
Based on learnings, this improves debuggability.
🔎 Proposed fix
except Exception as e:
- logger.error(f"Error validating chat config: {e}")
+ logger.exception("Error validating chat config")
return {"valid": False, "error": f"Validation error: {str(e)}"}🧰 Tools
🪛 Ruff (0.14.10)
555-555: Do not catch blind exception: Exception
(BLE001)
556-556: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
557-557: Use explicit conversion flag
Replace with conversion flag
(RUF010)
🤖 Prompt for AI Agents
In backends/advanced/src/advanced_omi_backend/controllers/system_controller.py
around lines 555-557, the exception handling logs the error with logger.error
which loses the stack trace; change the call to logger.exception(...) (or
logging.exception if using module logger) to log the full traceback and keep
returning the same {"valid": False, "error": ...} payload so diagnostics include
the stack trace while behavior remains unchanged.
Comment on lines
+23
to
+43
| const loadChatConfig = async () => { | ||
| setLoading(true) | ||
| setError('') | ||
| setMessage('') | ||
|
|
||
| try { | ||
| const response = await systemApi.getChatConfigRaw() | ||
| setConfigYaml(response.data.config_yaml || response.data) | ||
| setMessage('Configuration loaded successfully') | ||
| setTimeout(() => setMessage(''), 3000) | ||
| } catch (err: any) { | ||
| const status = err.response?.status | ||
| if (status === 401) { | ||
| setError('Unauthorized: admin privileges required') | ||
| } else { | ||
| setError(err.response?.data?.error || 'Failed to load configuration') | ||
| } | ||
| } finally { | ||
| setLoading(false) | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Find the backend chat config endpoint
fd -e "py" -e "tsx" -e "ts" | head -20
git ls-files | grep -E "(test|spec)" | head -10Repository: chronicler-ai/chronicle
Length of output: 1202
🏁 Script executed:
# Search for the chat config endpoint in Python backend
rg "get_chat_config|/api/admin/chat/config" --type py -A 10Repository: chronicler-ai/chronicle
Length of output: 3093
🏁 Script executed:
# Search for tests at line 160 area
rg "config_yaml|plain text" --type py --type js --type ts -B 2 -A 2Repository: chronicler-ai/chronicle
Length of output: 13610
🏁 Script executed:
# Search for tests related to chat config
rg "get_chat_config|chat.*config" --type py -i -B 3 -A 3 | grep -E "(test|def |assert|status)" | head -30Repository: chronicler-ai/chronicle
Length of output: 1365
🏁 Script executed:
# Find test files and check for chat config testing
fd -e "test_*.py" -o -e "*_test.py" | xargs rg "get_chat_config|/admin/chat" -lRepository: chronicler-ai/chronicle
Length of output: 437
🏁 Script executed:
# Check the robot test file for chat config tests
cat tests/endpoints/system_admin_tests.robot | head -200Repository: chronicler-ai/chronicle
Length of output: 8882
🏁 Script executed:
# Check the API service definition to see how getChatConfigRaw is implemented
cat backends/advanced/webui/src/services/api.ts | grep -A 10 "getChatConfigRaw"Repository: chronicler-ai/chronicle
Length of output: 480
Simplify response handling to match actual plain-text response format.
Line 30 accesses response.data.config_yaml || response.data, but the backend endpoint returns plain text directly with media_type="text/plain". The response.data.config_yaml condition will never be true. Simplify to just setConfigYaml(response.data).
Backend endpoint returns plain text
async def get_chat_config(current_user: User = Depends(current_superuser)):
"""Get chat configuration as YAML. Admin only."""
try:
yaml_content = await system_controller.get_chat_config_yaml()
return Response(content=yaml_content, media_type="text/plain")
Tests confirm this: ${prompt}= Set Variable ${response.text} (using .text for plain text, not .json())
🤖 Prompt for AI Agents
In backends/advanced/webui/src/components/ChatSettings.tsx around lines 23 to
43, the response handling assumes a JSON object with a config_yaml field but the
backend returns plain text; replace the conditional access with a direct
assignment of the response body to config YAML. Concretely, change
setConfigYaml(response.data.config_yaml || response.data) to simply
setConfigYaml(response.data), and leave the surrounding loading/message/error
handling intact.
Comment on lines
+185
to
+190
| <ul className="text-sm text-blue-700 dark:text-blue-300 space-y-1 list-disc list-inside"> | ||
| <li>Define the AI assistant's behavior and personality</li> | ||
| <li>Use YAML multi-line format (|) for readable prompts</li> | ||
| <li>Changes take effect immediately (no restart required)</li> | ||
| <li>Keep prompts clear and focused for best results</li> | ||
| </ul> |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, find and examine the ChatSettings.tsx file
find . -name "ChatSettings.tsx" -type fRepository: chronicler-ai/chronicle
Length of output: 124
🏁 Script executed:
# Find chat_service.py and model_registry.py
find . -name "chat_service.py" -o -name "model_registry.py" | head -20Repository: chronicler-ai/chronicle
Length of output: 190
🏁 Script executed:
# Search for API endpoints that handle system prompt
rg -n "system.*prompt|system_prompt" --type py -A 3 | head -50Repository: chronicler-ai/chronicle
Length of output: 4970
🏁 Script executed:
# Get full implementation of _get_system_prompt()
sed -n '146,170p' backends/advanced/src/advanced_omi_backend/chat_service.pyRepository: chronicler-ai/chronicle
Length of output: 1227
🏁 Script executed:
# Get full system prompt API endpoints
sed -n '460,530p' backends/advanced/src/advanced_omi_backend/controllers/system_controller.pyRepository: chronicler-ai/chronicle
Length of output: 2605
🏁 Script executed:
# Check model_registry.py for caching
cat backends/advanced/src/advanced_omi_backend/model_registry.py | head -100Repository: chronicler-ai/chronicle
Length of output: 3605
🏁 Script executed:
# Search for get_models_registry usage in chat_service.py
rg -n "get_models_registry" backends/advanced/src/advanced_omi_backend/chat_service.py -A 2 -B 2Repository: chronicler-ai/chronicle
Length of output: 589
🏁 Script executed:
# Check the load_models_config implementation with force_reload
rg -n "def load_models_config" backends/advanced/src/advanced_omi_backend/model_registry.py -A 20Repository: chronicler-ai/chronicle
Length of output: 924
🏁 Script executed:
# Check get_models_registry implementation
rg -n "def get_models_registry" backends/advanced/src/advanced_omi_backend/model_registry.py -A 10Repository: chronicler-ai/chronicle
Length of output: 479
🏁 Script executed:
# Look for any caching mechanism in model_registry
rg -n "_registry|_config|cache" backends/advanced/src/advanced_omi_backend/model_registry.py | head -30Repository: chronicler-ai/chronicle
Length of output: 786
Clarify help text regarding system prompt format.
The help text at line 187 is misleading: "Use YAML multi-line format (|) for readable prompts" suggests users should employ YAML syntax. However, the system prompt is stored as plain text within a YAML config file—users simply enter plain text without needing to consider YAML formatting conventions. Rephrase to "Enter clear, well-formatted text for the system prompt."
The statement "Changes take effect immediately (no restart required)" is accurate: save_chat_config_yaml() calls load_models_config(force_reload=True) to reload the config in memory after saving.
🤖 Prompt for AI Agents
In backends/advanced/webui/src/components/ChatSettings.tsx around lines 185 to
190, the help text is misleading about YAML formatting; replace the line "Use
YAML multi-line format (|) for readable prompts" with "Enter clear,
well-formatted text for the system prompt" (since the system prompt is stored as
plain text in a YAML file and users should enter plain text), and leave the
"Changes take effect immediately (no restart required)" line as-is because
save_chat_config_yaml() triggers load_models_config(force_reload=True). Ensure
punctuation and list styling remain consistent.
Comment on lines
+298
to
+308
| if recreate: | ||
| # Full recreation: down + up (fixes bind mount issues) | ||
| if not run_compose_command(service_name, 'down'): | ||
| console.print(f"[red]❌ Failed to stop {service_name}[/red]") | ||
| continue | ||
|
|
||
| if run_compose_command(service_name, 'up'): | ||
| console.print(f"[green]✅ {service_name} restarted[/green]") | ||
| success_count += 1 | ||
| else: | ||
| console.print(f"[red]❌ Failed to start {service_name}[/red]") |
Contributor
There was a problem hiding this comment.
Add safeguard or warning when service left in down state.
In the recreate path, if down succeeds (line 300) but up fails (line 304), the service remains stopped with no automatic recovery. This could leave critical services unavailable.
Consider one of these approaches:
- Attempt to restart the service using the old quick-restart method as a fallback
- Add explicit logging that clearly indicates the service is now down and requires manual intervention
- Track and report all services left in down state at the end of the operation
🔎 Example: Enhanced error handling with fallback
if recreate:
# Full recreation: down + up (fixes bind mount issues)
if not run_compose_command(service_name, 'down'):
console.print(f"[red]❌ Failed to stop {service_name}[/red]")
continue
if run_compose_command(service_name, 'up'):
console.print(f"[green]✅ {service_name} restarted[/green]")
success_count += 1
else:
- console.print(f"[red]❌ Failed to start {service_name}[/red]")
+ console.print(f"[red]❌ Failed to start {service_name} - attempting quick restart as fallback[/red]")
+ if run_compose_command(service_name, 'restart'):
+ console.print(f"[yellow]⚠️ {service_name} recovered using quick restart[/yellow]")
+ success_count += 1
+ else:
+ console.print(f"[red]💥 {service_name} is now DOWN and requires manual intervention[/red]")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if recreate: | |
| # Full recreation: down + up (fixes bind mount issues) | |
| if not run_compose_command(service_name, 'down'): | |
| console.print(f"[red]❌ Failed to stop {service_name}[/red]") | |
| continue | |
| if run_compose_command(service_name, 'up'): | |
| console.print(f"[green]✅ {service_name} restarted[/green]") | |
| success_count += 1 | |
| else: | |
| console.print(f"[red]❌ Failed to start {service_name}[/red]") | |
| if recreate: | |
| # Full recreation: down + up (fixes bind mount issues) | |
| if not run_compose_command(service_name, 'down'): | |
| console.print(f"[red]❌ Failed to stop {service_name}[/red]") | |
| continue | |
| if run_compose_command(service_name, 'up'): | |
| console.print(f"[green]✅ {service_name} restarted[/green]") | |
| success_count += 1 | |
| else: | |
| console.print(f"[red]❌ Failed to start {service_name} - attempting quick restart as fallback[/red]") | |
| if run_compose_command(service_name, 'restart'): | |
| console.print(f"[yellow]⚠️ {service_name} recovered using quick restart[/yellow]") | |
| success_count += 1 | |
| else: | |
| console.print(f"[red]💥 {service_name} is now DOWN and requires manual intervention[/red]") |
🤖 Prompt for AI Agents
In services.py around lines 298 to 308, when recreate succeeds in running `down`
but the subsequent `up` fails the service is left stopped with no indication or
recovery; add a safeguard: after a failed `up` attempt, immediately try the
existing quick-restart path as a fallback (call the quick restart
function/command), if that also fails record the service name in a new
`left_down` list and emit an explicit error log/warning stating the service is
currently down and requires manual intervention; finally, after the loop, report
a summary of services in `left_down` so operators can act.
|
| Metric | Count |
|---|---|
| ✅ Passed | 93 |
| ❌ Failed | 2 |
| 📊 Total | 95 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
|
| Metric | Count |
|---|---|
| ✅ Passed | 93 |
| ❌ Failed | 2 |
| 📊 Total | 95 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
…ion logging - Updated `start.sh` to improve shutdown handling by explicitly killing the backend process if running. - Modified `chat_service.py` to enhance logging for loading chat system prompts, providing clearer feedback on configuration usage. - Added a new `chat` field in `model_registry.py` for better chat service configuration management. - Updated vector store query parameters in `vector_stores.py` for improved clarity and functionality. - Enhanced the chat component in the web UI to conditionally auto-scroll based on message sending status.
|
| Metric | Count |
|---|---|
| ✅ Passed | 93 |
| ❌ Failed | 2 |
| 📊 Total | 95 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
|
| Metric | Count |
|---|---|
| ✅ Passed | 92 |
| ❌ Failed | 3 |
| 📊 Total | 95 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
|
| Metric | Count |
|---|---|
| ✅ Passed | 94 |
| ❌ Failed | 1 |
| 📊 Total | 95 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
|
| Metric | Count |
|---|---|
| ✅ Passed | 94 |
| ❌ Failed | 1 |
| 📊 Total | 95 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI Agents
In @backends/advanced/src/advanced_omi_backend/routers/modules/system_routes.py:
- Around line 169-176: Replace the current except block that logs the error and
raises HTTPException so it uses logger.exception(...) to record the full stack
trace and chain the raised HTTPException from the original exception;
specifically, in the try/except around awaiting request.body(), decoding
yaml_str and calling system_controller.validate_chat_config_yaml(yaml_str),
change logger.error(f"Failed to validate chat config: {e}") to
logger.exception("Failed to validate chat config") and change raise
HTTPException(status_code=500, detail=str(e)) to raise
HTTPException(status_code=500, detail=str(e)) from e.
- Around line 151-160: The exception handlers in the save-chat-config route
should preserve exception context and log full trace: in the ValueError except
block re-raise HTTPException(status_code=400, detail=str(e)) using "raise ...
from e", and in the general except block replace logger.error(...) with
logging.exception("Failed to save chat config") (or logger.exception if logger
is the module logger) and re-raise HTTPException(status_code=500, detail=str(e))
using "raise ... from e"; apply these changes around the request.body()/yaml_str
handling and the call to system_controller.save_chat_config_yaml to preserve
traceback and follow project patterns.
- Around line 137-142: Replace the broad logger.error call and bare raise with
logging.exception to record the full traceback and re-raise the HTTPException
chained to the original exception; specifically, in the except block catching
exceptions from system_controller.get_chat_config_yaml(), call
logger.exception("Failed to get chat config") and then raise
HTTPException(status_code=500, detail=str(e)) from e so the context is
preserved.
🧹 Nitpick comments (1)
backends/advanced/start.sh (1)
13-14: LGTM! Improved shutdown semantics.The targeted termination of
BACKEND_PIDwith proper error handling is a solid improvement over the previous approach. The conditional check and error suppression ensure robust cleanup.Optional: Consider signal consistency
For consistency with lines 65-68, you could either:
- Use explicit
-TERMthroughout (lines 65-68), or- Rely on the default SIGTERM in both places
Since
killdefaults to SIGTERM, the current behavior is identical, but explicit signals improve readability.[ -n "$AUDIO_WORKER_1_PID" ] && kill $AUDIO_WORKER_1_PID 2>/dev/null || true [ -n "$AUDIO_WORKER_2_PID" ] && kill $AUDIO_WORKER_2_PID 2>/dev/null || true [ -n "$RQ_WORKER_PID" ] && kill $RQ_WORKER_PID 2>/dev/null || true -[ -n "$BACKEND_PID" ] && kill $BACKEND_PID 2>/dev/null || true +[ -n "$BACKEND_PID" ] && kill -TERM $BACKEND_PID 2>/dev/null || true
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
backends/advanced/docker-compose-test.ymlbackends/advanced/src/advanced_omi_backend/chat_service.pybackends/advanced/src/advanced_omi_backend/model_registry.pybackends/advanced/src/advanced_omi_backend/routers/modules/system_routes.pybackends/advanced/src/advanced_omi_backend/services/memory/providers/vector_stores.pybackends/advanced/start.shbackends/advanced/webui/src/pages/Chat.tsxtests/endpoints/system_admin_tests.robot
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/endpoints/system_admin_tests.robot
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{py,ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{py,ts,tsx,js,jsx}: ALL imports must be at the top of the file after the docstring. NEVER import modules in the middle of functions or files.
Use lazy imports sparingly and only when absolutely necessary for circular import issues.
Group imports in order: standard library, third-party, local imports.
Always raise errors, never silently ignore them. Use explicit error handling with proper exceptions rather than silent failures.
Files:
backends/advanced/src/advanced_omi_backend/model_registry.pybackends/advanced/src/advanced_omi_backend/services/memory/providers/vector_stores.pybackends/advanced/webui/src/pages/Chat.tsxbackends/advanced/src/advanced_omi_backend/chat_service.pybackends/advanced/src/advanced_omi_backend/routers/modules/system_routes.py
**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.py: Do NOT add defensivehasattr()checks. Instead, research and understand input/response or class structure.
Use Black formatter with 100-character line length for Python code.
Use isort for Python import organization.
Use Wyoming protocol (JSONL + binary) for WebSocket communication for structured audio sessions.
Use job-based tracking system for all audio processing (WebSocket and file uploads) with stage events (audio → transcription → memory) and completion status.
Conversations must only be created when speech is detected, eliminating noise-only sessions from user interfaces.
Always store audio sessions inaudio_chunkscollection byaudio_uuid. Only create entries inconversationscollection when speech is detected.
Use email-based authentication with MongoDB ObjectId user IDs.
Use automatic{objectid_suffix}-{device_name}format for client registration.
All data must be scoped by user_id with efficient permission checking for data isolation.
JWT tokens are required for all endpoints and WebSocket connections.
Support pluggable memory providers (Chronicle native with individual facts or OpenMemory MCP delegation).
Files:
backends/advanced/src/advanced_omi_backend/model_registry.pybackends/advanced/src/advanced_omi_backend/services/memory/providers/vector_stores.pybackends/advanced/src/advanced_omi_backend/chat_service.pybackends/advanced/src/advanced_omi_backend/routers/modules/system_routes.py
**/*.{py,sh,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Use
uv run python3instead ofpython3for running Python commands, as uv is the package manager for all Python projects.
Files:
backends/advanced/src/advanced_omi_backend/model_registry.pybackends/advanced/src/advanced_omi_backend/services/memory/providers/vector_stores.pybackends/advanced/start.shbackends/advanced/src/advanced_omi_backend/chat_service.pybackends/advanced/src/advanced_omi_backend/routers/modules/system_routes.py
**/docker-compose*.{yml,yaml}
📄 CodeRabbit inference engine (CLAUDE.md)
**/docker-compose*.{yml,yaml}: Use Docker Compose without--no-cacheby default for faster builds. Only use--no-cachewhen explicitly needed for troubleshooting or when cached layers are causing issues.
Check if src/ is volume mounted. If not, dodocker compose buildso code changes are reflected. Do not simply rundocker compose restartas it will not rebuild the image.
Files:
backends/advanced/docker-compose-test.yml
🧠 Learnings (1)
📚 Learning: 2026-01-06T08:43:14.315Z
Learnt from: CR
Repo: SimpleOpenSoftware/chronicle PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-06T08:43:14.315Z
Learning: Applies to **/docker-compose*.{yml,yaml} : Check if src/ is volume mounted. If not, do `docker compose build` so code changes are reflected. Do not simply run `docker compose restart` as it will not rebuild the image.
Applied to files:
backends/advanced/docker-compose-test.yml
🧬 Code graph analysis (2)
backends/advanced/src/advanced_omi_backend/chat_service.py (1)
backends/advanced/src/advanced_omi_backend/model_registry.py (1)
get_models_registry(350-365)
backends/advanced/src/advanced_omi_backend/routers/modules/system_routes.py (1)
backends/advanced/src/advanced_omi_backend/controllers/system_controller.py (3)
get_chat_config_yaml(462-487)save_chat_config_yaml(490-537)validate_chat_config_yaml(540-557)
🪛 Ruff (0.14.10)
backends/advanced/src/advanced_omi_backend/chat_service.py
162-162: Do not catch blind exception: Exception
(BLE001)
backends/advanced/src/advanced_omi_backend/routers/modules/system_routes.py
135-135: Unused function argument: current_user
(ARG001)
135-135: Do not perform function call Depends in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
140-140: Do not catch blind exception: Exception
(BLE001)
141-141: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
142-142: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
148-148: Unused function argument: current_user
(ARG001)
148-148: Do not perform function call Depends in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
157-157: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
158-158: Do not catch blind exception: Exception
(BLE001)
159-159: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
160-160: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
166-166: Unused function argument: current_user
(ARG001)
166-166: Do not perform function call Depends in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
174-174: Do not catch blind exception: Exception
(BLE001)
175-175: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
176-176: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: robot-tests
🔇 Additional comments (6)
backends/advanced/webui/src/pages/Chat.tsx (2)
54-64: LGTM! Auto-scroll logic improved.The conditional auto-scroll is a good UX enhancement. It now scrolls only during active message sending or streaming, preventing unwanted scrolling when loading historical messages or switching sessions.
294-296: LGTM! Layout constraints are well-structured.The height classes create a proper full-height chat interface with viewport constraints. The
h-fullandmax-h-screencombination ensures the chat area fills available space without exceeding the viewport, while the sidebar'smax-h-screenmaintains consistent scrolling behavior.backends/advanced/src/advanced_omi_backend/services/memory/providers/vector_stores.py (1)
172-206: Qdrant API migration appears correct; verify response structure matches implementation.The migration to
query_points()with parameter namequeryis confirmed in the official Qdrant Python Client API. However, the response structure handling (response.points,result.score,result.payload) requires validation against the actual Qdrant version in use, since the web documentation snippet does not explicitly detail these field names.Before merging, confirm:
- That
response.pointsreturns an iterable of point objects with the expected fields- That
result.scorecontains the similarity score (not a different attribute name)- That
result.payloadis accessible as shownThe error handling currently returns an empty list on failure, which could silently hide API incompatibilities. Consider logging the full exception details for debugging this migration.
backends/advanced/docker-compose-test.yml (1)
17-17: LGTM! Config writability enables admin configuration management.Making config.yml writable is necessary for the admin chat configuration feature. Both the backend and workers services correctly have write access, ensuring configuration changes are properly propagated.
Also applies to: 163-163
backends/advanced/src/advanced_omi_backend/model_registry.py (1)
188-191: LGTM! Chat configuration properly integrated into model registry.The chat field addition follows the established pattern for configuration sections (memory, speaker_recognition) with proper Pydantic validation, sensible defaults, and clean integration into the registry loading flow.
Also applies to: 325-325, 344-345
backends/advanced/src/advanced_omi_backend/chat_service.py (1)
146-171: LGTM! System prompt configuration with proper fallback.The
_get_system_prompt()method provides clean integration with the model registry and implements graceful degradation. The broad exception handling (line 162) is acceptable here since it:
- Logs the failure with a warning
- Falls back to a sensible default prompt
- Ensures the chat service continues to function
The informative logging (✅ success,
⚠️ fallback) aids in debugging and operations.Also applies to: 423-424
Comment on lines
+137
to
+142
| try: | ||
| yaml_content = await system_controller.get_chat_config_yaml() | ||
| return Response(content=yaml_content, media_type="text/plain") | ||
| except Exception as e: | ||
| logger.error(f"Failed to get chat config: {e}") | ||
| raise HTTPException(status_code=500, detail=str(e)) |
Contributor
There was a problem hiding this comment.
Improve exception handling to follow project patterns.
The exception handling should use logging.exception() to capture the full stack trace and chain the raised exception with from e to preserve context.
🔎 Proposed fix
try:
yaml_content = await system_controller.get_chat_config_yaml()
return Response(content=yaml_content, media_type="text/plain")
except Exception as e:
- logger.error(f"Failed to get chat config: {e}")
- raise HTTPException(status_code=500, detail=str(e))
+ logger.exception("Failed to get chat config")
+ raise HTTPException(status_code=500, detail=str(e)) from e🧰 Tools
🪛 Ruff (0.14.10)
140-140: Do not catch blind exception: Exception
(BLE001)
141-141: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
142-142: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
🤖 Prompt for AI Agents
In @backends/advanced/src/advanced_omi_backend/routers/modules/system_routes.py
around lines 137 - 142, Replace the broad logger.error call and bare raise with
logging.exception to record the full traceback and re-raise the HTTPException
chained to the original exception; specifically, in the except block catching
exceptions from system_controller.get_chat_config_yaml(), call
logger.exception("Failed to get chat config") and then raise
HTTPException(status_code=500, detail=str(e)) from e so the context is
preserved.
Comment on lines
+151
to
+160
| try: | ||
| yaml_content = await request.body() | ||
| yaml_str = yaml_content.decode('utf-8') | ||
| result = await system_controller.save_chat_config_yaml(yaml_str) | ||
| return JSONResponse(content=result) | ||
| except ValueError as e: | ||
| raise HTTPException(status_code=400, detail=str(e)) | ||
| except Exception as e: | ||
| logger.error(f"Failed to save chat config: {e}") | ||
| raise HTTPException(status_code=500, detail=str(e)) |
Contributor
There was a problem hiding this comment.
Improve exception handling to follow project patterns.
Both exception handlers should chain exceptions with from e to preserve context, and the general exception handler should use logging.exception() instead of logger.error().
🔎 Proposed fix
try:
yaml_content = await request.body()
yaml_str = yaml_content.decode('utf-8')
result = await system_controller.save_chat_config_yaml(yaml_str)
return JSONResponse(content=result)
except ValueError as e:
- raise HTTPException(status_code=400, detail=str(e))
+ raise HTTPException(status_code=400, detail=str(e)) from e
except Exception as e:
- logger.error(f"Failed to save chat config: {e}")
- raise HTTPException(status_code=500, detail=str(e))
+ logger.exception("Failed to save chat config")
+ raise HTTPException(status_code=500, detail=str(e)) from e🧰 Tools
🪛 Ruff (0.14.10)
157-157: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
158-158: Do not catch blind exception: Exception
(BLE001)
159-159: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
160-160: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
🤖 Prompt for AI Agents
In @backends/advanced/src/advanced_omi_backend/routers/modules/system_routes.py
around lines 151 - 160, The exception handlers in the save-chat-config route
should preserve exception context and log full trace: in the ValueError except
block re-raise HTTPException(status_code=400, detail=str(e)) using "raise ...
from e", and in the general except block replace logger.error(...) with
logging.exception("Failed to save chat config") (or logger.exception if logger
is the module logger) and re-raise HTTPException(status_code=500, detail=str(e))
using "raise ... from e"; apply these changes around the request.body()/yaml_str
handling and the call to system_controller.save_chat_config_yaml to preserve
traceback and follow project patterns.
Comment on lines
+169
to
+176
| try: | ||
| yaml_content = await request.body() | ||
| yaml_str = yaml_content.decode('utf-8') | ||
| result = await system_controller.validate_chat_config_yaml(yaml_str) | ||
| return JSONResponse(content=result) | ||
| except Exception as e: | ||
| logger.error(f"Failed to validate chat config: {e}") | ||
| raise HTTPException(status_code=500, detail=str(e)) |
Contributor
There was a problem hiding this comment.
Improve exception handling to follow project patterns.
The exception handler should use logging.exception() instead of logger.error() and chain the raised exception with from e.
🔎 Proposed fix
try:
yaml_content = await request.body()
yaml_str = yaml_content.decode('utf-8')
result = await system_controller.validate_chat_config_yaml(yaml_str)
return JSONResponse(content=result)
except Exception as e:
- logger.error(f"Failed to validate chat config: {e}")
- raise HTTPException(status_code=500, detail=str(e))
+ logger.exception("Failed to validate chat config")
+ raise HTTPException(status_code=500, detail=str(e)) from e🧰 Tools
🪛 Ruff (0.14.10)
174-174: Do not catch blind exception: Exception
(BLE001)
175-175: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
176-176: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
🤖 Prompt for AI Agents
In @backends/advanced/src/advanced_omi_backend/routers/modules/system_routes.py
around lines 169 - 176, Replace the current except block that logs the error and
raises HTTPException so it uses logger.exception(...) to record the full stack
trace and chain the raised HTTPException from the original exception;
specifically, in the try/except around awaiting request.body(), decoding
yaml_str and calling system_controller.validate_chat_config_yaml(yaml_str),
change logger.error(f"Failed to validate chat config: {e}") to
logger.exception("Failed to validate chat config") and change raise
HTTPException(status_code=500, detail=str(e)) to raise
HTTPException(status_code=500, detail=str(e)) from e.
🎉 Robot Framework Test ResultsStatus: ✅ All tests passed!
📊 View ReportsGitHub Pages (Live Reports): Download Artifacts:
|
Collaborator
Author
|
@0xrushi the tests pass |
0xrushi
approved these changes
Jan 7, 2026
AnkushMalaker
added a commit
that referenced
this pull request
Feb 19, 2026
* audio upload extension with gdrive credentials
* FIX: API parameters
* UPDATE: tmp files cleanup n code refactored as per review
* REFACTOR: minor refactor as per review
* REFACTOR: minor update as per review
* UPDATE: gdrive sync logic
* REFACTOR: code update as per gdrive and update credential client
* REFACTOR: validation updated - as per review from CR
* UPDATE: code has been refactore for UUID for diffrent audio upload sources
* REFACTOR: updated code as per review
* Update documentation and configuration to reflect the transition from 'friend-backend' to 'chronicle-backend' across various files, including setup instructions, Docker configurations, and service logs.
* Update test script to use docker-compose-test.yml for all test-related operations
* Added standard MIT license
* Fix/cleanup model (#219)
* refactor memory
* add config
* docstring
* more cleanup
* code quality
* code quality
* unused return
* DOTTED GET
* Refactor Docker and CI configurations
- Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process.
- Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services.
- Added new dependencies for Google API clients in `uv.lock` to support upcoming features.
* Update configuration files for model providers and Docker setup
- Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram.
- Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving.
- Updated memory configuration endpoint to accept plain text for YAML input.
* Update transcription job handling to format speaker IDs
- Changed variable name from `speaker_name` to `speaker_id` for clarity.
- Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling.
* Remove loading of backend .env file in test environment setup
- Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests.
* Enhance configuration management and setup wizard
- Updated README to reflect the new setup wizard process.
- Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers.
- Improved user feedback during configuration updates, including success messages for configuration file updates.
- Enabled backup of existing `config.yml` before saving changes.
* Enhance HTTPS configuration in setup wizard
- Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates.
- Improved user prompts for server IP/domain input during HTTPS setup.
- Updated default behavior to use existing IP or localhost based on user input.
- Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access.
* Add source parameter to audio file writing in websocket controller
- Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking.
---------
Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
* fix/broken-tests (#230)
* refactor memory
* add config
* docstring
* more cleanup
* code quality
* code quality
* unused return
* DOTTED GET
* Refactor Docker and CI configurations
- Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process.
- Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services.
- Added new dependencies for Google API clients in `uv.lock` to support upcoming features.
* Update configuration files for model providers and Docker setup
- Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram.
- Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving.
- Updated memory configuration endpoint to accept plain text for YAML input.
* Update transcription job handling to format speaker IDs
- Changed variable name from `speaker_name` to `speaker_id` for clarity.
- Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling.
* Remove loading of backend .env file in test environment setup
- Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests.
* Enhance configuration management and setup wizard
- Updated README to reflect the new setup wizard process.
- Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers.
- Improved user feedback during configuration updates, including success messages for configuration file updates.
- Enabled backup of existing `config.yml` before saving changes.
* Enhance HTTPS configuration in setup wizard
- Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates.
- Improved user prompts for server IP/domain input during HTTPS setup.
- Updated default behavior to use existing IP or localhost based on user input.
- Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access.
* Add source parameter to audio file writing in websocket controller
- Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking.
* Refactor error handling in system controller and update memory config routes
- Replaced ValueError with HTTPException for better error handling in `save_diarization_settings` and `validate_memory_config` functions.
- Introduced a new Pydantic model, `MemoryConfigRequest`, for validating memory configuration requests in the system routes.
- Updated the `validate_memory_config` endpoint to accept the new request model, improving input handling and validation.
---------
Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
* Feat/add obsidian 3 (#233)
* obsidian support
* neo4j comment
* cleanup code
* unused line
* unused line
* Fix MemoryEntry object usage in chat service
* comment
* feat(obsidian): add obsidian memory search integration to chat
* unit test
* use rq
* neo4j service
* typefix
* test fix
* cleanup
* cleanup
* version changes
* profile
* remove unused imports
* Refactor memory configuration validation endpoints
- Removed the deprecated `validate_memory_config_raw` endpoint and replaced it with a new endpoint that accepts plain text for validation.
- Updated the existing `validate_memory_config` endpoint to clarify that it now accepts JSON input.
- Adjusted the API call in the frontend to point to the new validation endpoint.
* Refactor health check model configuration loading
- Updated the health check function to load model configuration from the models registry instead of the root config.
- Improved error handling by logging warnings when model configuration loading fails.
---------
Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
* Update .gitignore to exclude all files in app/ios and app/android directories (#238)
* fix: Copy full source code in speaker-recognition Dockerfile (#243)
Adds COPY src/ src/ step after dependency installation to ensure
all source files are available in the Docker image. This improves
build caching while ensuring complete source code is present.
* Enhance configuration management and add new setup scripts (#235)
* Enhance configuration management and add new setup scripts
- Updated .gitignore to include config.yml and its template.
- Added config.yml.template for default configuration settings.
- Introduced restart.sh script for service management.
- Enhanced services.py to load config.yml and check for Obsidian/Neo4j integration.
- Updated wizard.py to prompt for Obsidian/Neo4j configuration during setup and create config.yml from template if it doesn't exist.
* Refactor transcription providers and enhance configuration management
- Updated Docker Compose files to include the new Neo4j service configuration.
- Added support for Obsidian/Neo4j integration in the setup process.
- Refactored transcription providers to utilize a registry-driven approach for Deepgram and Parakeet.
- Enhanced error handling and logging in transcription processes.
- Improved environment variable management in test scripts to prioritize command-line overrides.
- Removed deprecated Parakeet provider implementation and streamlined audio stream workers.
* Update configuration management and enhance file structure, add test-matrix (#237)
* Update configuration management and enhance file structure
- Refactored configuration file paths to use a dedicated `config/` directory, including updates to `config.yml` and its template.
- Modified service scripts to load the new configuration path for `config.yml`.
- Enhanced `.gitignore` to include the new configuration files and templates.
- Updated documentation to reflect changes in configuration file locations and usage.
- Improved setup scripts to ensure proper creation and management of configuration files.
- Added new test configurations for various provider combinations to streamline testing processes.
* Add test requirements and clean up imports in wizard.py
- Introduced a new `test-requirements.txt` file to manage testing dependencies.
- Removed redundant import of `shutil` in `wizard.py` to improve code clarity.
* Add ConfigManager for unified configuration management
- Introduced a new `config_manager.py` module to handle reading and writing configurations from `config.yml` and `.env` files, ensuring backward compatibility.
- Refactored `ChronicleSetup` in `backends/advanced/init.py` to utilize `ConfigManager` for loading and updating configurations, simplifying the setup process.
- Removed redundant methods for loading and saving `config.yml` directly in `ChronicleSetup`, as these are now managed by `ConfigManager`.
- Enhanced user feedback during configuration updates, including success messages for changes made to configuration files.
* Refactor transcription provider configuration and enhance setup process
- Updated `.env.template` to clarify speech-to-text configuration and removed deprecated options for Mistral.
- Modified `docker-compose.yml` to streamline environment variable management by removing unused Mistral keys.
- Enhanced `ChronicleSetup` in `init.py` to provide clearer user feedback and updated the transcription provider selection process to rely on `config.yml`.
- Improved error handling in the websocket controller to determine the transcription provider from the model registry instead of environment variables.
- Updated health check routes to reflect the new method of retrieving the transcription provider from `config.yml`.
- Adjusted `config.yml.template` to include comments on transcription provider options for better user guidance.
* Enhance ConfigManager with deep merge functionality
- Updated the `update_memory_config` method to perform a deep merge of updates into the memory configuration, ensuring nested dictionaries are merged correctly.
- Added a new `_deep_merge` method to handle recursive merging of dictionaries, improving configuration management capabilities.
* Refactor run-test.sh and enhance memory extraction tests
- Removed deprecated environment variable handling for TRANSCRIPTION_PROVIDER in `run-test.sh`, streamlining the configuration process.
- Introduced a new `run-custom.sh` script for executing Robot tests with custom configurations, improving test flexibility.
- Enhanced memory extraction tests in `audio_keywords.robot` and `memory_keywords.robot` to include detailed assertions and result handling.
- Updated `queue_keywords.robot` to fail fast if a job is in a 'failed' state when expecting 'completed', improving error handling.
- Refactored `test_env.py` to load environment variables with correct precedence, ensuring better configuration management.
* unify tests to robot test, add some more clean up
* Update health check configuration in docker-compose-test.yml (#241)
- Increased the number of retries from 5 to 10 for improved resilience during service readiness checks.
- Extended the start period from 30s to 60s to allow more time for services to initialize before health checks commence.
* Add step to create test configuration file in robot-tests.yml
- Introduced a new step in the GitHub Actions workflow to copy the test configuration file from tests/configs/deepgram-openai.yml to a new config/config.yml.
- Added logging to confirm the creation of the test config file, improving visibility during the test setup process.
* remove cache step since not required
* coderabbit comments
* Refactor ConfigManager error handling for configuration file loading
- Updated the ConfigManager to raise RuntimeError exceptions when the configuration file is not found or is invalid, improving error visibility and user guidance.
- Removed fallback behavior that previously returned the current directory, ensuring users are explicitly informed about missing or invalid configuration files.
* Refactor _find_repo_root method in ConfigManager
- Updated the _find_repo_root method to locate the repository root using the __file__ location instead of searching for config/config.yml, simplifying the logic and improving reliability.
- Removed the previous error handling that raised a RuntimeError if the configuration file was not found, as the new approach assumes config_manager.py is always at the repo root.
* Enhance speaker recognition service integration and error handling (#245)
* Enhance speaker recognition service integration and error handling
- Updated `docker-compose-test.yml` to enable speaker recognition in the test environment and added a new `speaker-service-test` service for testing purposes.
- Refactored `run-test.sh` to improve the execution of Robot Framework tests from the repository root.
- Enhanced error handling in `speaker_recognition_client.py` to return detailed error messages for connection issues.
- Improved error logging in `speaker_jobs.py` to handle and report errors from the speaker recognition service more effectively.
- Updated `Dockerfile` to copy the full source code after dependencies are cached, ensuring all necessary files are included in the image.
* Remove integration tests workflow and enhance robot tests with HF_TOKEN verification
- Deleted the `integration-tests.yml` workflow file to streamline CI processes.
- Updated `robot-tests.yml` to include verification for the new `HF_TOKEN` secret, ensuring all required secrets are checked before running tests.
* Fix key access in system admin tests to use string indexing for speakers data
* Refactor Robot Framework tests and enhance error handling in memory services
- Removed the creation of the test environment file from the GitHub Actions workflow to streamline setup.
- Updated the Robot Framework tests to utilize a unified test script for improved consistency.
- Enhanced error messages in the MemoryService class to provide more context on connection failures for LLM and vector store providers.
- Added critical checks for API key presence in the OpenAIProvider class to ensure valid credentials are provided before proceeding.
- Adjusted various test setup scripts to use a centralized BACKEND_DIR variable for better maintainability and clarity.
* Refactor test container cleanup in run-robot-tests.sh
- Updated the script to dynamically construct container names from docker-compose services, improving maintainability and reducing hardcoded values.
- Enhanced the cleanup process for stuck test containers by utilizing the COMPOSE_PROJECT_NAME variable.
* Enhance run-robot-tests.sh for improved logging and cleanup
- Set absolute paths for consistent directory references to simplify navigation.
- Capture container logs, status, and resource usage for better debugging.
- Refactor cleanup process to utilize dynamic backend directory references, improving maintainability.
- Ensure proper navigation back to the tests directory after operations.
* Add speaker recognition configuration and update test script defaults
- Introduced speaker recognition settings in config.yml.template, allowing for easy enable/disable and service URL configuration.
- Updated run-robot-tests.sh to use a test-specific configuration file that disables speaker recognition for improved CI performance.
- Modified deepgram-openai.yml to disable speaker recognition during CI tests to enhance execution speed.
* Refactor speaker recognition configuration management
- Updated docker-compose-test.yml to clarify speaker recognition settings, now controlled via config.yml for improved CI performance.
- Enhanced model_registry.py to include a dedicated speaker_recognition field for better configuration handling.
- Modified speaker_recognition_client.py to load configuration from config.yml, allowing for dynamic enabling/disabling of the speaker recognition service based on the configuration.
* Add minimum worker count verification to infrastructure tests
- Introduced a new keyword to verify that the minimum number of workers are registered, enhancing the robustness of health checks.
- Updated the worker count validation test to include a wait mechanism for worker registration, improving test reliability.
- Clarified comments regarding expected worker counts to reflect the distinction between RQ and audio stream workers.
* Update configuration management and enhance model handling
- Added OBSIDIAN_ENABLED configuration to ChronicleSetup for improved feature toggling.
- Introduced speaker_recognition configuration handling in model_registry.py to streamline model loading.
- Refactored imports in deepgram.py to improve clarity and reduce redundancy.
* Refactor configuration management in wizard and ChronicleSetup (#246)
* Refactor configuration management in wizard and ChronicleSetup
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling.
- Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth.
- Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
* Fix string formatting for error message in ChronicleSetup
* added JWT issuers for audience auth for service interop and shared us… (#250)
* added JWT issuers for audience auth for service interop and shared user accounts
* amended default value in line wioth code
* Feat/edit chat system prompt (#247)
* Refactor configuration management in wizard and ChronicleSetup
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling.
- Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth.
- Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
* Fix string formatting for error message in ChronicleSetup
* Enhance chat configuration management and UI integration
- Updated `services.py` to allow service restart with an option to recreate containers, addressing WSL2 bind mount issues.
- Added new chat configuration management functions in `system_controller.py` for loading, saving, and validating chat prompts.
- Introduced `ChatSettings` component in the web UI for admin users to manage chat configurations easily.
- Updated API service methods in `api.ts` to support chat configuration endpoints.
- Integrated chat settings into the system management page for better accessibility.
* Refactor backend shutdown process and enhance chat service configuration logging
- Updated `start.sh` to improve shutdown handling by explicitly killing the backend process if running.
- Modified `chat_service.py` to enhance logging for loading chat system prompts, providing clearer feedback on configuration usage.
- Added a new `chat` field in `model_registry.py` for better chat service configuration management.
- Updated vector store query parameters in `vector_stores.py` for improved clarity and functionality.
- Enhanced the chat component in the web UI to conditionally auto-scroll based on message sending status.
* Return JSONResponse instead of raw result
* Refactor headers creation in system admin tests
* Make config.yml writable for admin updates
* Docs consolidation (#257)
* Enhance setup documentation and convenience scripts
- Updated the interactive setup wizard instructions to recommend using the convenience script `./wizard.sh` for easier configuration.
- Added detailed instructions for uploading and processing existing audio files via the API, including example commands for single and multiple file uploads.
- Introduced a new section on HAVPE relay configuration for ESP32 audio streaming, providing environment variable setup and command examples.
- Clarified the distributed deployment setup, including GPU and backend separation instructions, and added benefits of using Tailscale for networking.
- Removed outdated `getting-started.md` and `SETUP_SCRIPTS.md` files to streamline documentation and avoid redundancy.
* Update setup instructions and enhance service management scripts
- Replaced direct command instructions with convenience scripts (`./wizard.sh` and `./start.sh`) for easier setup and service management.
- Added detailed usage of convenience scripts for checking service status, restarting, and stopping services.
- Clarified the distinction between convenience scripts and direct command usage for improved user guidance.
* Update speaker recognition models and documentation
- Changed the speaker diarization model from `pyannote/speaker-diarization-3.1` to `pyannote/speaker-diarization-community-1` across multiple files for consistency.
- Updated README files to reflect the new model and its usage instructions, ensuring users have the correct links and information for setup.
- Enhanced clarity in configuration settings related to speaker recognition.
* Docs consolidation (#258)
* Enhance setup documentation and convenience scripts
- Updated the interactive setup wizard instructions to recommend using the convenience script `./wizard.sh` for easier configuration.
- Added detailed instructions for uploading and processing existing audio files via the API, including example commands for single and multiple file uploads.
- Introduced a new section on HAVPE relay configuration for ESP32 audio streaming, providing environment variable setup and command examples.
- Clarified the distributed deployment setup, including GPU and backend separation instructions, and added benefits of using Tailscale for networking.
- Removed outdated `getting-started.md` and `SETUP_SCRIPTS.md` files to streamline documentation and avoid redundancy.
* Update setup instructions and enhance service management scripts
- Replaced direct command instructions with convenience scripts (`./wizard.sh` and `./start.sh`) for easier setup and service management.
- Added detailed usage of convenience scripts for checking service status, restarting, and stopping services.
- Clarified the distinction between convenience scripts and direct command usage for improved user guidance.
* Update speaker recognition models and documentation
- Changed the speaker diarization model from `pyannote/speaker-diarization-3.1` to `pyannote/speaker-diarization-community-1` across multiple files for consistency.
- Updated README files to reflect the new model and its usage instructions, ensuring users have the correct links and information for setup.
- Enhanced clarity in configuration settings related to speaker recognition.
* Enhance transcription provider selection and update HTTPS documentation
- Added a new function in `wizard.py` to prompt users for their preferred transcription provider, allowing options for Deepgram, Parakeet ASR, or none.
- Updated the service setup logic to automatically include ASR services if Parakeet is selected.
- Introduced a new documentation file on SSL certificates and HTTPS setup, detailing the importance of HTTPS for secure connections and microphone access.
- Removed outdated HTTPS setup documentation from `backends/advanced/Docs/HTTPS_SETUP.md` to streamline resources.
* Remove HTTPS setup scripts and related configurations
- Deleted `init-https.sh`, `setup-https.sh`, and `nginx.conf.template` as part of the transition to a new HTTPS setup process.
- Updated `README.md` to reflect the new automatic HTTPS configuration via the setup wizard.
- Adjusted `init.py` to remove references to the deleted HTTPS scripts and ensure proper handling of Caddyfile generation for SSL.
- Streamlined documentation to clarify the new approach for HTTPS setup and configuration management.
* Update quickstart.md (#268)
* v0.2 (#279)
* Refactor configuration management in wizard and ChronicleSetup
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling.
- Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth.
- Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
* Fix string formatting for error message in ChronicleSetup
* Enhance chat configuration management and UI integration
- Updated `services.py` to allow service restart with an option to recreate containers, addressing WSL2 bind mount issues.
- Added new chat configuration management functions in `system_controller.py` for loading, saving, and validating chat prompts.
- Introduced `ChatSettings` component in the web UI for admin users to manage chat configurations easily.
- Updated API service methods in `api.ts` to support chat configuration endpoints.
- Integrated chat settings into the system management page for better accessibility.
* Refactor backend shutdown process and enhance chat service configuration logging
- Updated `start.sh` to improve shutdown handling by explicitly killing the backend process if running.
- Modified `chat_service.py` to enhance logging for loading chat system prompts, providing clearer feedback on configuration usage.
- Added a new `chat` field in `model_registry.py` for better chat service configuration management.
- Updated vector store query parameters in `vector_stores.py` for improved clarity and functionality.
- Enhanced the chat component in the web UI to conditionally auto-scroll based on message sending status.
* Implement plugin system for enhanced functionality and configuration management
- Introduced a new plugin architecture to allow for extensibility in the Chronicle application.
- Added Home Assistant plugin for controlling devices via natural language commands triggered by wake words.
- Implemented plugin configuration management endpoints in the API for loading, saving, and validating plugin settings.
- Enhanced the web UI with a dedicated Plugins page for managing plugin configurations.
- Updated Docker Compose files to include Tailscale integration for remote service access.
- Refactored existing services to support plugin interactions during conversation and memory processing.
- Improved error handling and logging for plugin initialization and execution processes.
* Enhance configuration management and plugin system integration
- Updated .gitignore to include plugins.yml for security reasons.
- Modified start.sh to allow passing additional arguments during service startup.
- Refactored wizard.py to support new HF_TOKEN configuration prompts and improved handling of wake words in plugin settings.
- Introduced a new setup_hf_token_if_needed function to streamline Hugging Face token management.
- Enhanced the GitHub Actions workflow to create plugins.yml from a template, ensuring proper configuration setup.
- Added detailed comments and documentation in the plugins.yml.template for better user guidance on Home Assistant integration.
* Implement Redis integration for client-user mapping and enhance wake word processing
- Added asynchronous Redis support in ClientManager for tracking client-user relationships.
- Introduced `initialize_redis_for_client_manager` to set up Redis for cross-container mapping.
- Updated `create_client_state` to use asynchronous tracking for client-user relationships.
- Enhanced wake word processing in PluginRouter with normalization and command extraction.
- Refactored DeepgramStreamingConsumer to utilize async Redis lookups for user ID retrieval.
- Set TTL on Redis streams during client state cleanup for better resource management.
* Refactor Deepgram worker management and enhance text normalization
- Disabled the batch Deepgram worker in favor of the streaming worker to prevent race conditions.
- Updated text normalization in wake word processing to replace punctuation with spaces, preserving word boundaries.
- Enhanced regex pattern for wake word matching to allow optional punctuation and whitespace after the last part.
- Improved logging in DeepgramStreamingConsumer for better visibility of message processing and error handling.
* Add original prompt retrieval and restoration in chat configuration test
- Implemented retrieval of the original chat prompt before saving a custom prompt to ensure test isolation.
- Added restoration of the original prompt after the test to prevent interference with subsequent tests.
- Enhanced the test documentation for clarity on the purpose of these changes.
* Refactor test execution and enhance documentation for integration tests
- Simplified test execution commands in CLAUDE.md and quickstart.md for better usability.
- Added instructions for running tests from the project root and clarified the process for executing the complete Robot Framework test suite.
- Introduced a new Docker service for the Deepgram streaming worker in docker-compose-test.yml to improve testing capabilities.
- Updated system_admin_tests.robot to use a defined default prompt for restoration, enhancing test reliability and clarity.
* Enhance test environment cleanup and improve Deepgram worker management
- Updated `run-test.sh` and `run-robot-tests.sh` to improve cleanup processes, including handling permission issues with Docker.
- Introduced a new function `mark_session_complete` in `session_controller.py` to ensure atomic updates for session completion status.
- Refactored WebSocket and conversation job handling to utilize the new session completion function, enhancing reliability.
- Updated `start-workers.sh` to enable the batch Deepgram worker alongside the streaming worker for improved transcription capabilities.
- Enhanced test scripts to verify the status of Deepgram workers and ensure proper cleanup of test containers.
* Refactor worker management and introduce orchestrator for improved process handling
- Replaced the bash-based `start-workers.sh` script with a Python-based worker orchestrator for better process management and health monitoring.
- Updated `docker-compose.yml` to configure the new orchestrator and adjust worker definitions, including the addition of audio persistence and stream workers.
- Enhanced the Dockerfile to remove the old startup script and ensure the orchestrator is executable.
- Introduced new modules for orchestrator configuration, health monitoring, process management, and worker registry to streamline worker lifecycle management.
- Improved environment variable handling for worker configuration and health checks.
* oops
* oops2
* Remove legacy test runner script and update worker orchestration
- Deleted the `run-test.sh` script, which was used for local test execution.
- Updated Docker configurations to replace the `start-workers.sh` script with `worker_orchestrator.py` for improved worker management.
- Enhanced health monitoring and process management in the orchestrator to ensure better reliability and logging.
- Adjusted deployment configurations to reflect the new orchestrator setup.
* Add bulk restart mechanism for RQ worker registration loss
- Introduced a new method `_handle_registration_loss` to manage RQ worker registration loss, replicating the behavior of the previous bash script.
- Implemented a cooldown period to prevent frequent restarts during network issues.
- Added logging for bulk restart actions and their outcomes to enhance monitoring and debugging capabilities.
- Created a `_restart_all_rq_workers` method to facilitate the bulk restart of RQ workers, ensuring they re-register with Redis upon startup.
* Enhance plugin architecture with event-driven system and test integration
- Introduced a new Test Event Plugin to log all plugin events to an SQLite database for integration testing.
- Updated the plugin system to utilize event subscriptions instead of access levels, allowing for more flexible event handling.
- Refactored the PluginRouter to dispatch events based on subscriptions, improving the event-driven architecture.
- Enhanced Docker configurations to support development and testing environments with appropriate dependencies.
- Added comprehensive integration tests to verify the functionality of the event dispatch system and plugin interactions.
- Updated documentation and test configurations to reflect the new event-based plugin structure.
* Enhance Docker configurations and startup script for test mode
- Updated `docker-compose-test.yml` to include a test command for services, enabling a dedicated test mode.
- Modified `start.sh` to support a `--test` flag, allowing the FastAPI backend to run with test-specific configurations.
- Adjusted worker commands to utilize the `--group test` option in test mode for improved orchestration and management.
* Refactor test scripts for improved reliability and clarity
- Updated `run-robot-tests.sh` to enhance the verification of the Deepgram batch worker process, ensuring non-numeric characters are removed from the check.
- Modified `plugin_tests.robot` to use a more explicit method for checking the length of subscriptions and added a skip condition for unavailable audio files.
- Adjusted `plugin_event_tests.robot` to load the test audio file from a variable, improving test data management.
- Refactored `plugin_keywords.robot` to utilize clearer length checks for subscriptions and event parts, enhancing readability and maintainability.
* remove mistral deadcode; notebooks untouched
* Refactor audio streaming endpoints and improve documentation
- Updated WebSocket endpoints to use a unified format with codec parameters (`/ws?codec=pcm` and `/ws?codec=opus`) for audio streaming, replacing the previous `/ws_pcm` and `/ws_omi` endpoints.
- Enhanced documentation to reflect the new endpoint structure and clarify audio processing capabilities.
- Removed deprecated audio cropping functionality and related configurations to streamline the audio processing workflow.
- Updated various components and scripts to align with the new endpoint structure, ensuring consistent usage across the application.
* Enhance testing infrastructure and API routes for plugin events
- Updated `docker-compose-test.yml` to introduce low speech detection thresholds for testing, improving the accuracy of speech detection during tests.
- Added new test-only API routes in `test_routes.py` for clearing and retrieving plugin events, ensuring a clean state between tests.
- Refactored existing test scripts to utilize the new API endpoints for event management, enhancing test reliability and clarity.
- Improved logging and error handling in various components to facilitate debugging during test execution.
- Adjusted environment variable handling in test setup scripts to streamline configuration and improve flexibility.
* Add audio pipeline architecture documentation and improve audio persistence worker configuration
- Introduced a comprehensive documentation file detailing the audio pipeline architecture, covering data flow, processing stages, and key components.
- Enhanced the audio persistence worker setup by implementing multiple concurrent workers to improve audio processing efficiency.
- Adjusted sleep intervals in the audio streaming persistence job for better responsiveness and event loop yielding.
- Updated test script to run the full suite of integration tests from the specified directory, ensuring thorough testing coverage.
* Add test container setup and teardown scripts
- Introduced `setup-test-containers.sh` for streamlined startup of test containers, including health checks and environment variable loading.
- Added `teardown-test-containers.sh` for simplified container shutdown, with options to remove volumes.
- Enhanced user feedback with color-coded messages for better visibility during test setup and teardown processes.
* Update worker count validation and websocket disconnect tests
- Adjusted worker count expectations in the Worker Count Validation Test to reflect an increase from 7 to 9 workers, accounting for additional audio persistence workers.
- Enhanced the WebSocket Disconnect Conversation End Reason Test by adding steps to maintain audio streaming during disconnection, ensuring accurate simulation of network dropout scenarios.
- Improved comments for clarity and added critical notes regarding inactivity timeout handling.
* Refactor audio storage to MongoDB chunks and enhance cleanup settings management
- Replaced the legacy AudioFile model with AudioChunkDocument for storing audio data in MongoDB, optimizing storage and retrieval.
- Introduced CleanupSettings dataclass for managing soft-deletion configurations, including auto-cleanup and retention days.
- Added admin API routes for retrieving and saving cleanup settings, ensuring better control over data retention policies.
- Updated audio processing workflows to utilize MongoDB chunks, removing dependencies on disk-based audio files.
- Enhanced tests to validate the new audio chunk storage and cleanup functionalities, ensuring robust integration with existing systems.
* Refactor audio processing to utilize MongoDB chunks and enhance job handling
- Removed audio file path parameters from various functions, transitioning to audio data retrieval from MongoDB chunks.
- Updated the `start_post_conversation_jobs` function to reflect changes in audio handling, ensuring jobs reconstruct audio from database chunks.
- Enhanced the `transcribe_full_audio_job` and `recognise_speakers_job` to process audio directly from memory, eliminating the need for temporary files.
- Improved error handling and logging for audio data retrieval, ensuring better feedback during processing.
- Added a new utility function for converting PCM data to WAV format in memory, streamlining audio format handling.
* Refactor speaker recognition client to use in-memory audio data
- Updated methods to accept audio data as bytes instead of file paths, enhancing performance by eliminating disk I/O.
- Improved logging to reflect in-memory audio processing, providing better insights during speaker identification and diarization.
- Streamlined audio data handling in the `diarize_identify_match` and `diarize_and_identify` methods, ensuring consistency across the client.
- Removed temporary file handling, simplifying the audio processing workflow and reducing potential file system errors.
* Add mock providers and update testing workflows for API-independent execution
- Introduced `MockLLMProvider` and `MockTranscriptionProvider` to facilitate testing without external API dependencies, allowing for consistent and controlled test environments.
- Created `run-no-api-tests.sh` script to execute tests that do not require API keys, ensuring separation of API-dependent and independent tests.
- Updated Robot Framework test configurations to utilize mock services, enhancing test reliability and reducing external dependencies.
- Modified existing test workflows to include new configurations and ensure proper handling of results for tests excluding API keys.
- Added `mock-services.yml` configuration to disable external API services while maintaining core functionality for testing purposes.
- Enhanced documentation to reflect the new tagging system for tests requiring API keys, improving clarity on test execution requirements.
* Enhance testing documentation and workflows for API key separation
- Updated CLAUDE.md to clarify test execution modes, emphasizing the separation of tests requiring API keys from those that do not.
- Expanded the testing guidelines in TESTING_GUIDELINES.md to detail the organization of tests based on API dependencies, including tagging conventions and execution paths.
- Improved mock-services.yml to include dummy configurations for LLM and embedding services, ensuring tests can run without actual API calls.
- Added comprehensive documentation on GitHub workflows for different test scenarios, enhancing clarity for contributors and maintainers.
* Update test configurations and documentation for API key management
- Modified `plugins.yml.template` to implement event subscriptions for the Home Assistant plugin, enhancing its event-driven capabilities.
- Revised `README.md` to clarify test execution processes, emphasizing the distinction between tests requiring API keys and those that do not.
- Updated `mock-services.yml` to streamline mock configurations, ensuring compatibility with the new testing workflows.
- Added `requires-api-keys` tags to relevant test cases across various test files, improving organization and clarity regarding API dependencies.
- Enhanced documentation for test scripts and configurations, providing clearer guidance for contributors on executing tests based on API key requirements.
* Add optional service profile to Docker Compose test configuration
* Refactor audio processing and job handling for transcription workflows
- Updated `upload_and_process_audio_files` and `start_post_conversation_jobs` to enqueue transcription jobs separately for file uploads, ensuring accurate processing order.
- Enhanced logging to provide clearer insights into job enqueuing and processing stages.
- Removed batch transcription from the post-conversation job chain for streaming audio, utilizing the streaming transcript directly.
- Introduced word-level timestamps in the `Conversation` model to improve transcript detail and accuracy.
- Updated tests to reflect changes in job handling and ensure proper verification of post-conversation processing.
* Remove unnecessary network aliases from speaker service in Docker Compose configuration
* Add network aliases for speaker service in Docker Compose configuration
* Refactor Conversation model to use string for provider field
- Updated the `Conversation` model to replace the `TranscriptProvider` enum with a string type for the `provider` field, allowing for greater flexibility in provider names.
- Adjusted related job functions to accommodate this change, simplifying provider handling in the transcription workflow.
* Enhance configuration and model handling for waveform data
- Updated Docker Compose files to mount the entire config directory, allowing for better management of configuration files.
- Introduced a new `WaveformData` model to store pre-computed waveform visualization data, improving UI performance by enabling waveform display without real-time decoding.
- Enhanced the `app_factory` and `job` models to include the new `WaveformData` model, ensuring proper initialization and data handling.
- Implemented waveform generation logic in a new worker module, allowing for on-demand waveform creation from audio chunks.
- Added API endpoints for retrieving and generating waveform data, improving the overall audio processing capabilities.
- Updated tests to cover new functionality and ensure robustness in waveform data handling.
* Add SDK testing scripts for authentication, conversation retrieval, and audio upload
- Introduced three new test scripts: `sdk_test_auth.py`, `sdk_test_conversations.py`, and `sdk_test_upload.py`.
- Each script tests different functionalities of the SDK, including authentication, conversation retrieval, and audio file uploads.
- The scripts utilize the `ChronicleClient` to perform operations and print results for verification.
- Enhanced testing capabilities for the SDK, ensuring robust validation of core features.
* Enhance audio processing and conversation handling for large files
- Added configuration options for speaker recognition chunking in `.env.template`, allowing for better management of large audio files.
- Updated `get_conversations` function to include an `include_deleted` parameter for filtering conversations based on their deletion status.
- Enhanced `finalize_session` method in `AudioStreamProducer` to send an end marker to Redis, ensuring proper session closure.
- Introduced `reconstruct_audio_segments` function to yield audio segments with overlap for efficient processing of lengthy conversations.
- Implemented merging of overlapping speaker segments to improve accuracy in speaker recognition.
- Added integration tests for WebSocket streaming transcription to validate the end_marker functionality and overall transcription flow.
* archive
* Implement annotation system and enhance audio processing capabilities
- Introduced a new annotation model to support user edits and AI-powered suggestions for memories and transcripts.
- Added annotation routes for CRUD operations, enabling the creation and management of annotations via the API.
- Enhanced the audio processing workflow to support fetching audio segments from the backend, improving speaker recognition accuracy.
- Updated the speaker recognition client to handle conversation-based audio fetching, allowing for better management of large audio files.
- Implemented a cron job for generating AI suggestions on potential errors in transcripts and memories, improving user experience and content accuracy.
- Enhanced the web UI to support inline editing of transcript segments and memory content, providing a more interactive user experience.
- Updated configuration files to support new features and improve overall system flexibility.
* Implement OmegaConf-based configuration management for backend settings
- Introduced a new configuration loader using OmegaConf for unified management of backend settings.
- Updated existing configuration functions to leverage the new loader, enhancing flexibility and maintainability.
- Added support for environment variable interpolation in configuration files.
- Refactored various components to retrieve settings from the new configuration system, improving consistency across the application.
- Updated requirements to include OmegaConf as a dependency.
- Enhanced documentation and comments for clarity on configuration management.
* Refactor .env.template and remove unused diarization configuration
- Updated the .env.template to clarify its purpose for secret values and streamline setup instructions.
- Removed the deprecated diarization_config.json.template file, as it is no longer needed.
- Added new environment variables for Langfuse and Tailscale integration to enhance observability and remote service access.
* Implement legacy environment variable syntax support in configuration loader
- Added custom OmegaConf resolvers to handle legacy ${VAR:-default} syntax for backward compatibility.
- Introduced a preprocessing function to convert legacy syntax in YAML files to OmegaConf-compatible format.
- Updated the load_config function to utilize the new preprocessing for loading defaults and user configurations.
- Enhanced documentation for clarity on the new legacy syntax handling.
* Add plugins configuration path retrieval and refactor usage
- Introduced a new function `get_plugins_yml_path` to centralize the retrieval of the plugins.yml file path.
- Updated `system_controller.py` and `plugin_service.py` to use the new function for improved maintainability and consistency in accessing the plugins configuration.
- Enhanced code clarity by removing hardcoded paths and utilizing the centralized configuration method.
* Unify plugin terminology and fix memory job dependencies
Plugin terminology: subscriptions→events, trigger→condition
Memory jobs: no longer blocked by disabled speaker recognition
* Update Docker Compose configuration and enhance system routes
- Updated Docker Compose files to mount the entire config directory, consolidating configuration management.
- Refactored the `save_diarization_settings` function to improve clarity and maintainability by renaming it to `save_diarization_settings_controller`.
- Enhanced the System component in the web UI to include configuration diagnostics, providing better visibility into system health and issues.
* circular import
* Refactor testing infrastructure and enhance container management
- Updated the testing documentation to reflect a new Makefile-based approach for running tests and managing containers.
- Introduced new scripts for container management, including starting, stopping, restarting, and cleaning containers while preserving logs.
- Added a cleanup script to handle data ownership and permissions correctly.
- Implemented a logging system that saves container logs automatically before cleanup.
- Enhanced the README with detailed instructions for running tests and managing the test environment.
* Add Email Summarizer Plugin and SMTP Email Service
- Introduced the Email Summarizer Plugin that automatically sends email summaries upon conversation completion.
- Implemented SMTP Email Service for sending emails, supporting HTML and plain text formats with TLS/SSL encryption.
- Added configuration options for SMTP settings in the .env.template and plugins.yml.template.
- Created comprehensive documentation for plugin development and usage, including a new plugin generation script.
- Enhanced testing coverage for the Email Summarizer Plugin and SMTP Email Service to ensure reliability and functionality.
* Refactor plugin management and introduce Email Summarizer setup
- Removed the static PLUGINS dictionary and replaced it with a dynamic discovery mechanism for plugins.
- Implemented a new setup process for plugins, allowing for configuration via individual setup scripts.
- Added the Email Summarizer plugin with a dedicated setup script for SMTP configuration.
- Enhanced the main setup flow to support community plugins and their configuration.
- Cleaned up unused functions related to plugin configuration and streamlined the overall plugin setup process.
* Enhance plugin configuration and documentation
- Updated the .env.template to include new configuration options for the Home Assistant and Email Summarizer plugins, including server URLs, tokens, and additional settings.
- Refactored Docker Compose files to correctly mount plugin configuration paths.
- Introduced comprehensive documentation for plugin configuration architecture, detailing the separation of concerns for orchestration, settings, and secrets.
- Added individual configuration files for the Home Assistant and Email Summarizer plugins, ensuring proper management of non-secret settings and environment variable references.
- Improved the plugin loading process to merge configurations from multiple sources, enhancing flexibility and maintainability.
* Refactor plugin setup process to allow interactive user input
- Updated the plugin setup script to run interactively, enabling plugins to prompt for user input during configuration.
- Removed output capturing to facilitate real-time interaction and improved error messaging to include exit codes for better debugging.
* Add shared setup utilities for interactive configuration
- Introduced `setup_utils.py` containing functions for reading environment variables, prompting user input, and masking sensitive values.
- Refactored existing code in `wizard.py` and `init.py` to utilize these shared utilities, improving code reuse and maintainability.
- Updated documentation to include usage examples for the new utilities in plugin setup scripts, enhancing developer experience and clarity.
* Enhance plugin security architecture and configuration management
- Introduced a three-file separation for plugin configuration to improve security:
- `backends/advanced/.env` for secrets (gitignored)
- `config/plugins.yml` for orchestration with environment variable references
- `plugins/{plugin_id}/config.yml` for non-secret defaults
- Updated documentation to emphasize the importance of using `${ENV_VAR}` syntax for sensitive data and provided examples of correct usage.
- Enhanced the Email Summarizer plugin setup process to automatically update `config/plugins.yml` with environment variable references, ensuring secrets are not hardcoded.
- Added new fields to the User model for notification email management and improved error logging in user-related functions.
- Refactored audio chunk utilities to use a consistent method for fetching conversation metadata.
* Refactor backend components for improved functionality and stability
- Added a new parameter `transcript_version_id` to the `open_conversation_job` function to support streaming transcript versioning.
- Enhanced error handling in `check_enrolled_speakers_job` and `recognise_speakers_job` to allow conversations to proceed even when the speaker service is unavailable, improving resilience.
- Updated `send_to_adv.py` to support dynamic WebSocket and HTTP protocols based on environment settings, enhancing configuration flexibility.
- Introduced a background task in `send_to_adv.py` to handle incoming messages from the backend, ensuring connection stability and logging interim results.
* Refactor plugin setup timing to enhance configuration flow
* Refactor save_diarization_settings_controller to improve validation and error handling
- Updated the controller to filter out invalid settings instead of raising an error for each unknown key, allowing for more flexible input.
- Added a check to reject requests with no valid settings provided, enhancing robustness.
- Adjusted logging to reflect the filtered settings being saved.
* Refactor audio processing and conversation management for improved deduplication and tracking
* Refactor audio and email handling for improved functionality and security
- Updated `mask_value` function to handle whitespace more effectively.
- Enhanced `create_plugin` to remove existing directories when using the `--force` option.
- Changed logging level from error to debug for existing admin user checks.
- Improved client ID generation logging for clarity.
- Removed unused fields from conversation creation.
- Added HTML escaping in email templates to prevent XSS attacks.
- Updated audio file download function to include user ID for better tracking.
- Adjusted WebSocket connection settings to respect SSL verification based on environment variables.
* Refactor audio upload functionality to remove unused parameters
- Removed `auto_generate_client` and `folder` parameters from audio upload functions to streamline the API.
- Updated related function calls and documentation to reflect these changes, enhancing clarity and reducing complexity.
* Refactor Email Summarizer plugin configuration for improved clarity and security
- Removed outdated migration instructions from `plugin-configuration.md` to streamline documentation.
- Enhanced `README.md` to clearly outline the three-file separation for plugin configuration, emphasizing the roles of `.env`, `config.yml`, and `plugins.yml`.
- Updated `setup.py` to reflect changes in orchestration settings, ensuring only relevant configurations are included in `config/plugins.yml`.
- Improved security messaging to highlight the importance of not committing secrets to version control.
* Update API key configuration in config.yml.template to use environment variable syntax for improved flexibility and security. This change standardizes the way API keys are referenced across different models and services. (#273)
Co-authored-by: roshan.john <roshanjohn1460@gmail.com>
* Refactor Redis job queue cleanup process for improved success tracking
- Replaced total job count with separate counters for successful and failed jobs during Redis queue cleanup.
- Enhanced logging to provide detailed feedback on the number of jobs cleared and any failures encountered.
- Improved error handling to ensure job counts are accurately reflected even when exceptions occur.
* fix tests
* Update CI workflows to use 'docker compose' for log retrieval and added container status check
- Replaced 'docker logs' commands with 'docker compose -f docker-compose-test.yml logs' for consistency across workflows.
- Added a check for running containers before saving logs to enhance debugging capabilities.
* test fixes
* FIX StreamingTranscriptionConsumer to support cumulative audio timestamp adjustments
- Added `audio_offset_seconds` to track cumulative audio duration for accurate timestamp adjustments across transcription sessions.
- Updated `store_final_result` method to adjust word and segment timestamps based on cumulative audio offset.
- Improved logging to reflect changes in audio offset after storing results.
- Modified Makefile and documentation to clarify test execution options, including new tags for slow and SDK tests, enhancing test organization and execution clarity.
* Enhance test container setup and improve error messages in integration tests
- Set `COMPOSE_PROJECT_NAME` for test containers to ensure consistent naming.
- Consolidated error messages in the `websocket_transcription_e2e_test.robot` file for clarity, improving readability and debugging.
* Improve WebSocket closing logic and enhance integration test teardown
- Added timeout handling for WebSocket closure in `AudioStreamClient` to prevent hanging and ensure clean disconnection.
- Updated integration tests to log the total chunks sent when closing audio streams, improving clarity on resource management during test teardown.
* Refactor job status handling to align with RQ standards
- Updated job status checks across various modules to use "started" and "finished" instead of "processing" and "completed" for consistency with RQ's naming conventions.
- Adjusted related logging and response messages to reflect the new status terminology.
- Simplified Docker Compose project name handling in test scripts to avoid conflicts and improve clarity in test environment setup.
* Update test configurations and improve audio inactivity handling
- Increased `SPEECH_INACTIVITY_THRESHOLD_SECONDS` to 20 seconds in `docker-compose-test.yml` for better audio duration handling during tests.
- Refactored session handling in `session_controller.py` to clarify client ID usage.
- Updated `conversation_utils.py` to track speech activity using audio timestamps, enhancing accuracy in inactivity detection.
- Simplified test scripts by removing unnecessary `COMPOSE_PROJECT_NAME` references, aligning with the new project naming convention.
- Adjusted integration tests to reflect changes in inactivity timeout and ensure proper handling of audio timestamps.
* Refactor audio processing and enhance error handling
- Updated `worker_orchestrator.py` to use `logger.exception` for improved error logging.
- Changed default MongoDB database name from "friend-lite" to "chronicle" in multiple files for consistency.
- Added a new method `close_stream_without_stop` in `audio_stream_client.py` to handle abrupt WebSocket disconnections.
- Enhanced audio validation in `audio_utils.py` to support automatic resampling of audio data if sample rates do not match.
- Improved logging in various modules to provide clearer insights during audio processing and event dispatching.
* Enhance Docker command handling and configuration management
- Updated `run_compose_command` to support separate build commands for services, including profile management for backend and speaker-recognition services.
- Improved error handling and output streaming during Docker command execution.
- Added `ensure_docker_network` function to verify and create the required Docker network before starting services.
- Refactored configuration files to utilize `oc.env` for environment variable management, ensuring better compatibility and flexibility across different environments.
* Enhance configuration loading to support custom config file paths
- Added support for the CONFIG_FILE environment variable to allow specifying custom configuration files for testing.
- Implemented logic to handle both absolute paths and relative filenames for the configuration file, improving flexibility in configuration management.
* Update test scripts to use TEST_CONFIG_FILE for configuration management
- Replaced CONFIG_FILE with TEST_CONFIG_FILE in both run-no-api-tests.sh and run-robot-tests.sh to standardize configuration file usage.
- Updated paths to point to mock and deepgram-openai configuration files inside the container, improving clarity and consistency in test setups.
* Refactor audio upload response handling and improve error reporting
- Updated `upload_and_process_audio_files` to return appropriate HTTP status codes based on upload results: 400 for all failures, 207 for partial successes, and 200 for complete success.
- Enhanced error messages in the audio upload tests to provide clearer feedback on upload failures, including specific error details for better debugging.
- Adjusted test scripts to ensure consistent handling of conversation IDs in job metadata, improving validation checks for job creation.
* Refactor audio processing and job handling to improve transcription management
- Updated `upload_and_process_audio_files` to check for transcription provider availability before enqueueing jobs, enhancing error handling and logging.
- Modified `start_post_conversation_jobs` to conditionally enqueue memory extraction jobs based on configuration, improving flexibility in job management.
- Enhanced event dispatch job dependencies to only include jobs that were actually enqueued, ensuring accurate job tracking.
- Added `is_transcription_available` function to check transcription provider status, improving modularity and clarity in the transcription workflow.
* Enhance integration tests for plugin events and improve error handling
- Updated integration tests to filter plugin events by conversation ID, ensuring accurate event tracking and reducing noise from fixture events.
- Improved error messages in event verification to include conversation ID context, enhancing clarity during test failures.
- Refactored audio upload handling to check for transcription job creation, allowing for more robust conversation polling and error reporting.
- Added new keyword to verify conversation end reasons, improving test coverage for conversation state validation.
* Enhance speaker recognition testing and audio processing
- Added mock speaker recognition client to facilitate testing without resource-intensive dependencies.
- Updated Docker Compose configurations to include mock speaker client for test environments.
- Refactored audio segment reconstruction to ensure precise clipping based on time boundaries.
- Improved error handling in transcription jobs and speaker recognition workflows to enhance robustness.
- Adjusted integration tests to utilize real-time pacing for audio chunk streaming, improving test accuracy.
* Refactor audio chunk retrieval and enhance logging in audio processing
- Introduced logging for audio chunk requests to improve traceability.
- Replaced manual audio chunk processing with a dedicated `reconstruct_audio_segment` function for better clarity and efficiency.
- Improved error handling during audio reconstruction to provide more informative responses in case of failures.
- Cleaned up imports and removed redundant code related to audio chunk calculations.
* Refactor mock speaker recognition client and improve testing structure
- Replaced direct import of mock client with a structured import from the new testing module.
- Introduced a dedicated `mock_speaker_client.py` to provide a mock implementation for speaker recognition, facilitating testing without heavy dependencies.
- Added an `__init__.py` file in the testing directory to organize testing utilities and mocks.
* Enhance conversation model to include word-level timestamps and improve transcript handling
- Added a new `words` field to the `Conversation` model for storing word-level timestamps.
- Updated methods to handle word data during transcript version creation, ensuring compatibility with speaker recognition.
- Refactored conversation job processing to utilize the new word structure, improving data integrity and access.
- Enhanced speaker recognition job to read words from the new standardized location, ensuring backward compatibility with legacy data.
* Implement speaker reprocessing feature and enhance timeout calculation
- Added a new endpoint to reprocess speaker identification for existing transcripts, creating a new version with re-identified speakers.
- Introduced a method to calculate proportional t…
AnkushMalaker
added a commit
that referenced
this pull request
Feb 19, 2026
* audio upload extension with gdrive credentials
* FIX: API parameters
* UPDATE: tmp files cleanup n code refactored as per review
* REFACTOR: minor refactor as per review
* REFACTOR: minor update as per review
* UPDATE: gdrive sync logic
* REFACTOR: code update as per gdrive and update credential client
* REFACTOR: validation updated - as per review from CR
* UPDATE: code has been refactore for UUID for diffrent audio upload sources
* REFACTOR: updated code as per review
* Update documentation and configuration to reflect the transition from 'friend-backend' to 'chronicle-backend' across various files, including setup instructions, Docker configurations, and service logs.
* Update test script to use docker-compose-test.yml for all test-related operations
* Added standard MIT license
* Fix/cleanup model (#219)
* refactor memory
* add config
* docstring
* more cleanup
* code quality
* code quality
* unused return
* DOTTED GET
* Refactor Docker and CI configurations
- Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process.
- Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services.
- Added new dependencies for Google API clients in `uv.lock` to support upcoming features.
* Update configuration files for model providers and Docker setup
- Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram.
- Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving.
- Updated memory configuration endpoint to accept plain text for YAML input.
* Update transcription job handling to format speaker IDs
- Changed variable name from `speaker_name` to `speaker_id` for clarity.
- Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling.
* Remove loading of backend .env file in test environment setup
- Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests.
* Enhance configuration management and setup wizard
- Updated README to reflect the new setup wizard process.
- Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers.
- Improved user feedback during configuration updates, including success messages for configuration file updates.
- Enabled backup of existing `config.yml` before saving changes.
* Enhance HTTPS configuration in setup wizard
- Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates.
- Improved user prompts for server IP/domain input during HTTPS setup.
- Updated default behavior to use existing IP or localhost based on user input.
- Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access.
* Add source parameter to audio file writing in websocket controller
- Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking.
---------
Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
* fix/broken-tests (#230)
* refactor memory
* add config
* docstring
* more cleanup
* code quality
* code quality
* unused return
* DOTTED GET
* Refactor Docker and CI configurations
- Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process.
- Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services.
- Added new dependencies for Google API clients in `uv.lock` to support upcoming features.
* Update configuration files for model providers and Docker setup
- Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram.
- Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving.
- Updated memory configuration endpoint to accept plain text for YAML input.
* Update transcription job handling to format speaker IDs
- Changed variable name from `speaker_name` to `speaker_id` for clarity.
- Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling.
* Remove loading of backend .env file in test environment setup
- Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests.
* Enhance configuration management and setup wizard
- Updated README to reflect the new setup wizard process.
- Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers.
- Improved user feedback during configuration updates, including success messages for configuration file updates.
- Enabled backup of existing `config.yml` before saving changes.
* Enhance HTTPS configuration in setup wizard
- Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates.
- Improved user prompts for server IP/domain input during HTTPS setup.
- Updated default behavior to use existing IP or localhost based on user input.
- Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access.
* Add source parameter to audio file writing in websocket controller
- Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking.
* Refactor error handling in system controller and update memory config routes
- Replaced ValueError with HTTPException for better error handling in `save_diarization_settings` and `validate_memory_config` functions.
- Introduced a new Pydantic model, `MemoryConfigRequest`, for validating memory configuration requests in the system routes.
- Updated the `validate_memory_config` endpoint to accept the new request model, improving input handling and validation.
---------
Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
* Feat/add obsidian 3 (#233)
* obsidian support
* neo4j comment
* cleanup code
* unused line
* unused line
* Fix MemoryEntry object usage in chat service
* comment
* feat(obsidian): add obsidian memory search integration to chat
* unit test
* use rq
* neo4j service
* typefix
* test fix
* cleanup
* cleanup
* version changes
* profile
* remove unused imports
* Refactor memory configuration validation endpoints
- Removed the deprecated `validate_memory_config_raw` endpoint and replaced it with a new endpoint that accepts plain text for validation.
- Updated the existing `validate_memory_config` endpoint to clarify that it now accepts JSON input.
- Adjusted the API call in the frontend to point to the new validation endpoint.
* Refactor health check model configuration loading
- Updated the health check function to load model configuration from the models registry instead of the root config.
- Improved error handling by logging warnings when model configuration loading fails.
---------
Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
* Update .gitignore to exclude all files in app/ios and app/android directories (#238)
* fix: Copy full source code in speaker-recognition Dockerfile (#243)
Adds COPY src/ src/ step after dependency installation to ensure
all source files are available in the Docker image. This improves
build caching while ensuring complete source code is present.
* Enhance configuration management and add new setup scripts (#235)
* Enhance configuration management and add new setup scripts
- Updated .gitignore to include config.yml and its template.
- Added config.yml.template for default configuration settings.
- Introduced restart.sh script for service management.
- Enhanced services.py to load config.yml and check for Obsidian/Neo4j integration.
- Updated wizard.py to prompt for Obsidian/Neo4j configuration during setup and create config.yml from template if it doesn't exist.
* Refactor transcription providers and enhance configuration management
- Updated Docker Compose files to include the new Neo4j service configuration.
- Added support for Obsidian/Neo4j integration in the setup process.
- Refactored transcription providers to utilize a registry-driven approach for Deepgram and Parakeet.
- Enhanced error handling and logging in transcription processes.
- Improved environment variable management in test scripts to prioritize command-line overrides.
- Removed deprecated Parakeet provider implementation and streamlined audio stream workers.
* Update configuration management and enhance file structure, add test-matrix (#237)
* Update configuration management and enhance file structure
- Refactored configuration file paths to use a dedicated `config/` directory, including updates to `config.yml` and its template.
- Modified service scripts to load the new configuration path for `config.yml`.
- Enhanced `.gitignore` to include the new configuration files and templates.
- Updated documentation to reflect changes in configuration file locations and usage.
- Improved setup scripts to ensure proper creation and management of configuration files.
- Added new test configurations for various provider combinations to streamline testing processes.
* Add test requirements and clean up imports in wizard.py
- Introduced a new `test-requirements.txt` file to manage testing dependencies.
- Removed redundant import of `shutil` in `wizard.py` to improve code clarity.
* Add ConfigManager for unified configuration management
- Introduced a new `config_manager.py` module to handle reading and writing configurations from `config.yml` and `.env` files, ensuring backward compatibility.
- Refactored `ChronicleSetup` in `backends/advanced/init.py` to utilize `ConfigManager` for loading and updating configurations, simplifying the setup process.
- Removed redundant methods for loading and saving `config.yml` directly in `ChronicleSetup`, as these are now managed by `ConfigManager`.
- Enhanced user feedback during configuration updates, including success messages for changes made to configuration files.
* Refactor transcription provider configuration and enhance setup process
- Updated `.env.template` to clarify speech-to-text configuration and removed deprecated options for Mistral.
- Modified `docker-compose.yml` to streamline environment variable management by removing unused Mistral keys.
- Enhanced `ChronicleSetup` in `init.py` to provide clearer user feedback and updated the transcription provider selection process to rely on `config.yml`.
- Improved error handling in the websocket controller to determine the transcription provider from the model registry instead of environment variables.
- Updated health check routes to reflect the new method of retrieving the transcription provider from `config.yml`.
- Adjusted `config.yml.template` to include comments on transcription provider options for better user guidance.
* Enhance ConfigManager with deep merge functionality
- Updated the `update_memory_config` method to perform a deep merge of updates into the memory configuration, ensuring nested dictionaries are merged correctly.
- Added a new `_deep_merge` method to handle recursive merging of dictionaries, improving configuration management capabilities.
* Refactor run-test.sh and enhance memory extraction tests
- Removed deprecated environment variable handling for TRANSCRIPTION_PROVIDER in `run-test.sh`, streamlining the configuration process.
- Introduced a new `run-custom.sh` script for executing Robot tests with custom configurations, improving test flexibility.
- Enhanced memory extraction tests in `audio_keywords.robot` and `memory_keywords.robot` to include detailed assertions and result handling.
- Updated `queue_keywords.robot` to fail fast if a job is in a 'failed' state when expecting 'completed', improving error handling.
- Refactored `test_env.py` to load environment variables with correct precedence, ensuring better configuration management.
* unify tests to robot test, add some more clean up
* Update health check configuration in docker-compose-test.yml (#241)
- Increased the number of retries from 5 to 10 for improved resilience during service readiness checks.
- Extended the start period from 30s to 60s to allow more time for services to initialize before health checks commence.
* Add step to create test configuration file in robot-tests.yml
- Introduced a new step in the GitHub Actions workflow to copy the test configuration file from tests/configs/deepgram-openai.yml to a new config/config.yml.
- Added logging to confirm the creation of the test config file, improving visibility during the test setup process.
* remove cache step since not required
* coderabbit comments
* Refactor ConfigManager error handling for configuration file loading
- Updated the ConfigManager to raise RuntimeError exceptions when the configuration file is not found or is invalid, improving error visibility and user guidance.
- Removed fallback behavior that previously returned the current directory, ensuring users are explicitly informed about missing or invalid configuration files.
* Refactor _find_repo_root method in ConfigManager
- Updated the _find_repo_root method to locate the repository root using the __file__ location instead of searching for config/config.yml, simplifying the logic and improving reliability.
- Removed the previous error handling that raised a RuntimeError if the configuration file was not found, as the new approach assumes config_manager.py is always at the repo root.
* Enhance speaker recognition service integration and error handling (#245)
* Enhance speaker recognition service integration and error handling
- Updated `docker-compose-test.yml` to enable speaker recognition in the test environment and added a new `speaker-service-test` service for testing purposes.
- Refactored `run-test.sh` to improve the execution of Robot Framework tests from the repository root.
- Enhanced error handling in `speaker_recognition_client.py` to return detailed error messages for connection issues.
- Improved error logging in `speaker_jobs.py` to handle and report errors from the speaker recognition service more effectively.
- Updated `Dockerfile` to copy the full source code after dependencies are cached, ensuring all necessary files are included in the image.
* Remove integration tests workflow and enhance robot tests with HF_TOKEN verification
- Deleted the `integration-tests.yml` workflow file to streamline CI processes.
- Updated `robot-tests.yml` to include verification for the new `HF_TOKEN` secret, ensuring all required secrets are checked before running tests.
* Fix key access in system admin tests to use string indexing for speakers data
* Refactor Robot Framework tests and enhance error handling in memory services
- Removed the creation of the test environment file from the GitHub Actions workflow to streamline setup.
- Updated the Robot Framework tests to utilize a unified test script for improved consistency.
- Enhanced error messages in the MemoryService class to provide more context on connection failures for LLM and vector store providers.
- Added critical checks for API key presence in the OpenAIProvider class to ensure valid credentials are provided before proceeding.
- Adjusted various test setup scripts to use a centralized BACKEND_DIR variable for better maintainability and clarity.
* Refactor test container cleanup in run-robot-tests.sh
- Updated the script to dynamically construct container names from docker-compose services, improving maintainability and reducing hardcoded values.
- Enhanced the cleanup process for stuck test containers by utilizing the COMPOSE_PROJECT_NAME variable.
* Enhance run-robot-tests.sh for improved logging and cleanup
- Set absolute paths for consistent directory references to simplify navigation.
- Capture container logs, status, and resource usage for better debugging.
- Refactor cleanup process to utilize dynamic backend directory references, improving maintainability.
- Ensure proper navigation back to the tests directory after operations.
* Add speaker recognition configuration and update test script defaults
- Introduced speaker recognition settings in config.yml.template, allowing for easy enable/disable and service URL configuration.
- Updated run-robot-tests.sh to use a test-specific configuration file that disables speaker recognition for improved CI performance.
- Modified deepgram-openai.yml to disable speaker recognition during CI tests to enhance execution speed.
* Refactor speaker recognition configuration management
- Updated docker-compose-test.yml to clarify speaker recognition settings, now controlled via config.yml for improved CI performance.
- Enhanced model_registry.py to include a dedicated speaker_recognition field for better configuration handling.
- Modified speaker_recognition_client.py to load configuration from config.yml, allowing for dynamic enabling/disabling of the speaker recognition service based on the configuration.
* Add minimum worker count verification to infrastructure tests
- Introduced a new keyword to verify that the minimum number of workers are registered, enhancing the robustness of health checks.
- Updated the worker count validation test to include a wait mechanism for worker registration, improving test reliability.
- Clarified comments regarding expected worker counts to reflect the distinction between RQ and audio stream workers.
* Update configuration management and enhance model handling
- Added OBSIDIAN_ENABLED configuration to ChronicleSetup for improved feature toggling.
- Introduced speaker_recognition configuration handling in model_registry.py to streamline model loading.
- Refactored imports in deepgram.py to improve clarity and reduce redundancy.
* Refactor configuration management in wizard and ChronicleSetup (#246)
* Refactor configuration management in wizard and ChronicleSetup
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling.
- Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth.
- Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
* Fix string formatting for error message in ChronicleSetup
* added JWT issuers for audience auth for service interop and shared us… (#250)
* added JWT issuers for audience auth for service interop and shared user accounts
* amended default value in line wioth code
* Feat/edit chat system prompt (#247)
* Refactor configuration management in wizard and ChronicleSetup
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling.
- Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth.
- Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
* Fix string formatting for error message in ChronicleSetup
* Enhance chat configuration management and UI integration
- Updated `services.py` to allow service restart with an option to recreate containers, addressing WSL2 bind mount issues.
- Added new chat configuration management functions in `system_controller.py` for loading, saving, and validating chat prompts.
- Introduced `ChatSettings` component in the web UI for admin users to manage chat configurations easily.
- Updated API service methods in `api.ts` to support chat configuration endpoints.
- Integrated chat settings into the system management page for better accessibility.
* Refactor backend shutdown process and enhance chat service configuration logging
- Updated `start.sh` to improve shutdown handling by explicitly killing the backend process if running.
- Modified `chat_service.py` to enhance logging for loading chat system prompts, providing clearer feedback on configuration usage.
- Added a new `chat` field in `model_registry.py` for better chat service configuration management.
- Updated vector store query parameters in `vector_stores.py` for improved clarity and functionality.
- Enhanced the chat component in the web UI to conditionally auto-scroll based on message sending status.
* Return JSONResponse instead of raw result
* Refactor headers creation in system admin tests
* Make config.yml writable for admin updates
* Docs consolidation (#257)
* Enhance setup documentation and convenience scripts
- Updated the interactive setup wizard instructions to recommend using the convenience script `./wizard.sh` for easier configuration.
- Added detailed instructions for uploading and processing existing audio files via the API, including example commands for single and multiple file uploads.
- Introduced a new section on HAVPE relay configuration for ESP32 audio streaming, providing environment variable setup and command examples.
- Clarified the distributed deployment setup, including GPU and backend separation instructions, and added benefits of using Tailscale for networking.
- Removed outdated `getting-started.md` and `SETUP_SCRIPTS.md` files to streamline documentation and avoid redundancy.
* Update setup instructions and enhance service management scripts
- Replaced direct command instructions with convenience scripts (`./wizard.sh` and `./start.sh`) for easier setup and service management.
- Added detailed usage of convenience scripts for checking service status, restarting, and stopping services.
- Clarified the distinction between convenience scripts and direct command usage for improved user guidance.
* Update speaker recognition models and documentation
- Changed the speaker diarization model from `pyannote/speaker-diarization-3.1` to `pyannote/speaker-diarization-community-1` across multiple files for consistency.
- Updated README files to reflect the new model and its usage instructions, ensuring users have the correct links and information for setup.
- Enhanced clarity in configuration settings related to speaker recognition.
* Docs consolidation (#258)
* Enhance setup documentation and convenience scripts
- Updated the interactive setup wizard instructions to recommend using the convenience script `./wizard.sh` for easier configuration.
- Added detailed instructions for uploading and processing existing audio files via the API, including example commands for single and multiple file uploads.
- Introduced a new section on HAVPE relay configuration for ESP32 audio streaming, providing environment variable setup and command examples.
- Clarified the distributed deployment setup, including GPU and backend separation instructions, and added benefits of using Tailscale for networking.
- Removed outdated `getting-started.md` and `SETUP_SCRIPTS.md` files to streamline documentation and avoid redundancy.
* Update setup instructions and enhance service management scripts
- Replaced direct command instructions with convenience scripts (`./wizard.sh` and `./start.sh`) for easier setup and service management.
- Added detailed usage of convenience scripts for checking service status, restarting, and stopping services.
- Clarified the distinction between convenience scripts and direct command usage for improved user guidance.
* Update speaker recognition models and documentation
- Changed the speaker diarization model from `pyannote/speaker-diarization-3.1` to `pyannote/speaker-diarization-community-1` across multiple files for consistency.
- Updated README files to reflect the new model and its usage instructions, ensuring users have the correct links and information for setup.
- Enhanced clarity in configuration settings related to speaker recognition.
* Enhance transcription provider selection and update HTTPS documentation
- Added a new function in `wizard.py` to prompt users for their preferred transcription provider, allowing options for Deepgram, Parakeet ASR, or none.
- Updated the service setup logic to automatically include ASR services if Parakeet is selected.
- Introduced a new documentation file on SSL certificates and HTTPS setup, detailing the importance of HTTPS for secure connections and microphone access.
- Removed outdated HTTPS setup documentation from `backends/advanced/Docs/HTTPS_SETUP.md` to streamline resources.
* Remove HTTPS setup scripts and related configurations
- Deleted `init-https.sh`, `setup-https.sh`, and `nginx.conf.template` as part of the transition to a new HTTPS setup process.
- Updated `README.md` to reflect the new automatic HTTPS configuration via the setup wizard.
- Adjusted `init.py` to remove references to the deleted HTTPS scripts and ensure proper handling of Caddyfile generation for SSL.
- Streamlined documentation to clarify the new approach for HTTPS setup and configuration management.
* Update quickstart.md (#268)
* v0.2 (#279)
* Refactor configuration management in wizard and ChronicleSetup
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling.
- Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth.
- Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
* Fix string formatting for error message in ChronicleSetup
* Enhance chat configuration management and UI integration
- Updated `services.py` to allow service restart with an option to recreate containers, addressing WSL2 bind mount issues.
- Added new chat configuration management functions in `system_controller.py` for loading, saving, and validating chat prompts.
- Introduced `ChatSettings` component in the web UI for admin users to manage chat configurations easily.
- Updated API service methods in `api.ts` to support chat configuration endpoints.
- Integrated chat settings into the system management page for better accessibility.
* Refactor backend shutdown process and enhance chat service configuration logging
- Updated `start.sh` to improve shutdown handling by explicitly killing the backend process if running.
- Modified `chat_service.py` to enhance logging for loading chat system prompts, providing clearer feedback on configuration usage.
- Added a new `chat` field in `model_registry.py` for better chat service configuration management.
- Updated vector store query parameters in `vector_stores.py` for improved clarity and functionality.
- Enhanced the chat component in the web UI to conditionally auto-scroll based on message sending status.
* Implement plugin system for enhanced functionality and configuration management
- Introduced a new plugin architecture to allow for extensibility in the Chronicle application.
- Added Home Assistant plugin for controlling devices via natural language commands triggered by wake words.
- Implemented plugin configuration management endpoints in the API for loading, saving, and validating plugin settings.
- Enhanced the web UI with a dedicated Plugins page for managing plugin configurations.
- Updated Docker Compose files to include Tailscale integration for remote service access.
- Refactored existing services to support plugin interactions during conversation and memory processing.
- Improved error handling and logging for plugin initialization and execution processes.
* Enhance configuration management and plugin system integration
- Updated .gitignore to include plugins.yml for security reasons.
- Modified start.sh to allow passing additional arguments during service startup.
- Refactored wizard.py to support new HF_TOKEN configuration prompts and improved handling of wake words in plugin settings.
- Introduced a new setup_hf_token_if_needed function to streamline Hugging Face token management.
- Enhanced the GitHub Actions workflow to create plugins.yml from a template, ensuring proper configuration setup.
- Added detailed comments and documentation in the plugins.yml.template for better user guidance on Home Assistant integration.
* Implement Redis integration for client-user mapping and enhance wake word processing
- Added asynchronous Redis support in ClientManager for tracking client-user relationships.
- Introduced `initialize_redis_for_client_manager` to set up Redis for cross-container mapping.
- Updated `create_client_state` to use asynchronous tracking for client-user relationships.
- Enhanced wake word processing in PluginRouter with normalization and command extraction.
- Refactored DeepgramStreamingConsumer to utilize async Redis lookups for user ID retrieval.
- Set TTL on Redis streams during client state cleanup for better resource management.
* Refactor Deepgram worker management and enhance text normalization
- Disabled the batch Deepgram worker in favor of the streaming worker to prevent race conditions.
- Updated text normalization in wake word processing to replace punctuation with spaces, preserving word boundaries.
- Enhanced regex pattern for wake word matching to allow optional punctuation and whitespace after the last part.
- Improved logging in DeepgramStreamingConsumer for better visibility of message processing and error handling.
* Add original prompt retrieval and restoration in chat configuration test
- Implemented retrieval of the original chat prompt before saving a custom prompt to ensure test isolation.
- Added restoration of the original prompt after the test to prevent interference with subsequent tests.
- Enhanced the test documentation for clarity on the purpose of these changes.
* Refactor test execution and enhance documentation for integration tests
- Simplified test execution commands in CLAUDE.md and quickstart.md for better usability.
- Added instructions for running tests from the project root and clarified the process for executing the complete Robot Framework test suite.
- Introduced a new Docker service for the Deepgram streaming worker in docker-compose-test.yml to improve testing capabilities.
- Updated system_admin_tests.robot to use a defined default prompt for restoration, enhancing test reliability and clarity.
* Enhance test environment cleanup and improve Deepgram worker management
- Updated `run-test.sh` and `run-robot-tests.sh` to improve cleanup processes, including handling permission issues with Docker.
- Introduced a new function `mark_session_complete` in `session_controller.py` to ensure atomic updates for session completion status.
- Refactored WebSocket and conversation job handling to utilize the new session completion function, enhancing reliability.
- Updated `start-workers.sh` to enable the batch Deepgram worker alongside the streaming worker for improved transcription capabilities.
- Enhanced test scripts to verify the status of Deepgram workers and ensure proper cleanup of test containers.
* Refactor worker management and introduce orchestrator for improved process handling
- Replaced the bash-based `start-workers.sh` script with a Python-based worker orchestrator for better process management and health monitoring.
- Updated `docker-compose.yml` to configure the new orchestrator and adjust worker definitions, including the addition of audio persistence and stream workers.
- Enhanced the Dockerfile to remove the old startup script and ensure the orchestrator is executable.
- Introduced new modules for orchestrator configuration, health monitoring, process management, and worker registry to streamline worker lifecycle management.
- Improved environment variable handling for worker configuration and health checks.
* oops
* oops2
* Remove legacy test runner script and update worker orchestration
- Deleted the `run-test.sh` script, which was used for local test execution.
- Updated Docker configurations to replace the `start-workers.sh` script with `worker_orchestrator.py` for improved worker management.
- Enhanced health monitoring and process management in the orchestrator to ensure better reliability and logging.
- Adjusted deployment configurations to reflect the new orchestrator setup.
* Add bulk restart mechanism for RQ worker registration loss
- Introduced a new method `_handle_registration_loss` to manage RQ worker registration loss, replicating the behavior of the previous bash script.
- Implemented a cooldown period to prevent frequent restarts during network issues.
- Added logging for bulk restart actions and their outcomes to enhance monitoring and debugging capabilities.
- Created a `_restart_all_rq_workers` method to facilitate the bulk restart of RQ workers, ensuring they re-register with Redis upon startup.
* Enhance plugin architecture with event-driven system and test integration
- Introduced a new Test Event Plugin to log all plugin events to an SQLite database for integration testing.
- Updated the plugin system to utilize event subscriptions instead of access levels, allowing for more flexible event handling.
- Refactored the PluginRouter to dispatch events based on subscriptions, improving the event-driven architecture.
- Enhanced Docker configurations to support development and testing environments with appropriate dependencies.
- Added comprehensive integration tests to verify the functionality of the event dispatch system and plugin interactions.
- Updated documentation and test configurations to reflect the new event-based plugin structure.
* Enhance Docker configurations and startup script for test mode
- Updated `docker-compose-test.yml` to include a test command for services, enabling a dedicated test mode.
- Modified `start.sh` to support a `--test` flag, allowing the FastAPI backend to run with test-specific configurations.
- Adjusted worker commands to utilize the `--group test` option in test mode for improved orchestration and management.
* Refactor test scripts for improved reliability and clarity
- Updated `run-robot-tests.sh` to enhance the verification of the Deepgram batch worker process, ensuring non-numeric characters are removed from the check.
- Modified `plugin_tests.robot` to use a more explicit method for checking the length of subscriptions and added a skip condition for unavailable audio files.
- Adjusted `plugin_event_tests.robot` to load the test audio file from a variable, improving test data management.
- Refactored `plugin_keywords.robot` to utilize clearer length checks for subscriptions and event parts, enhancing readability and maintainability.
* remove mistral deadcode; notebooks untouched
* Refactor audio streaming endpoints and improve documentation
- Updated WebSocket endpoints to use a unified format with codec parameters (`/ws?codec=pcm` and `/ws?codec=opus`) for audio streaming, replacing the previous `/ws_pcm` and `/ws_omi` endpoints.
- Enhanced documentation to reflect the new endpoint structure and clarify audio processing capabilities.
- Removed deprecated audio cropping functionality and related configurations to streamline the audio processing workflow.
- Updated various components and scripts to align with the new endpoint structure, ensuring consistent usage across the application.
* Enhance testing infrastructure and API routes for plugin events
- Updated `docker-compose-test.yml` to introduce low speech detection thresholds for testing, improving the accuracy of speech detection during tests.
- Added new test-only API routes in `test_routes.py` for clearing and retrieving plugin events, ensuring a clean state between tests.
- Refactored existing test scripts to utilize the new API endpoints for event management, enhancing test reliability and clarity.
- Improved logging and error handling in various components to facilitate debugging during test execution.
- Adjusted environment variable handling in test setup scripts to streamline configuration and improve flexibility.
* Add audio pipeline architecture documentation and improve audio persistence worker configuration
- Introduced a comprehensive documentation file detailing the audio pipeline architecture, covering data flow, processing stages, and key components.
- Enhanced the audio persistence worker setup by implementing multiple concurrent workers to improve audio processing efficiency.
- Adjusted sleep intervals in the audio streaming persistence job for better responsiveness and event loop yielding.
- Updated test script to run the full suite of integration tests from the specified directory, ensuring thorough testing coverage.
* Add test container setup and teardown scripts
- Introduced `setup-test-containers.sh` for streamlined startup of test containers, including health checks and environment variable loading.
- Added `teardown-test-containers.sh` for simplified container shutdown, with options to remove volumes.
- Enhanced user feedback with color-coded messages for better visibility during test setup and teardown processes.
* Update worker count validation and websocket disconnect tests
- Adjusted worker count expectations in the Worker Count Validation Test to reflect an increase from 7 to 9 workers, accounting for additional audio persistence workers.
- Enhanced the WebSocket Disconnect Conversation End Reason Test by adding steps to maintain audio streaming during disconnection, ensuring accurate simulation of network dropout scenarios.
- Improved comments for clarity and added critical notes regarding inactivity timeout handling.
* Refactor audio storage to MongoDB chunks and enhance cleanup settings management
- Replaced the legacy AudioFile model with AudioChunkDocument for storing audio data in MongoDB, optimizing storage and retrieval.
- Introduced CleanupSettings dataclass for managing soft-deletion configurations, including auto-cleanup and retention days.
- Added admin API routes for retrieving and saving cleanup settings, ensuring better control over data retention policies.
- Updated audio processing workflows to utilize MongoDB chunks, removing dependencies on disk-based audio files.
- Enhanced tests to validate the new audio chunk storage and cleanup functionalities, ensuring robust integration with existing systems.
* Refactor audio processing to utilize MongoDB chunks and enhance job handling
- Removed audio file path parameters from various functions, transitioning to audio data retrieval from MongoDB chunks.
- Updated the `start_post_conversation_jobs` function to reflect changes in audio handling, ensuring jobs reconstruct audio from database chunks.
- Enhanced the `transcribe_full_audio_job` and `recognise_speakers_job` to process audio directly from memory, eliminating the need for temporary files.
- Improved error handling and logging for audio data retrieval, ensuring better feedback during processing.
- Added a new utility function for converting PCM data to WAV format in memory, streamlining audio format handling.
* Refactor speaker recognition client to use in-memory audio data
- Updated methods to accept audio data as bytes instead of file paths, enhancing performance by eliminating disk I/O.
- Improved logging to reflect in-memory audio processing, providing better insights during speaker identification and diarization.
- Streamlined audio data handling in the `diarize_identify_match` and `diarize_and_identify` methods, ensuring consistency across the client.
- Removed temporary file handling, simplifying the audio processing workflow and reducing potential file system errors.
* Add mock providers and update testing workflows for API-independent execution
- Introduced `MockLLMProvider` and `MockTranscriptionProvider` to facilitate testing without external API dependencies, allowing for consistent and controlled test environments.
- Created `run-no-api-tests.sh` script to execute tests that do not require API keys, ensuring separation of API-dependent and independent tests.
- Updated Robot Framework test configurations to utilize mock services, enhancing test reliability and reducing external dependencies.
- Modified existing test workflows to include new configurations and ensure proper handling of results for tests excluding API keys.
- Added `mock-services.yml` configuration to disable external API services while maintaining core functionality for testing purposes.
- Enhanced documentation to reflect the new tagging system for tests requiring API keys, improving clarity on test execution requirements.
* Enhance testing documentation and workflows for API key separation
- Updated CLAUDE.md to clarify test execution modes, emphasizing the separation of tests requiring API keys from those that do not.
- Expanded the testing guidelines in TESTING_GUIDELINES.md to detail the organization of tests based on API dependencies, including tagging conventions and execution paths.
- Improved mock-services.yml to include dummy configurations for LLM and embedding services, ensuring tests can run without actual API calls.
- Added comprehensive documentation on GitHub workflows for different test scenarios, enhancing clarity for contributors and maintainers.
* Update test configurations and documentation for API key management
- Modified `plugins.yml.template` to implement event subscriptions for the Home Assistant plugin, enhancing its event-driven capabilities.
- Revised `README.md` to clarify test execution processes, emphasizing the distinction between tests requiring API keys and those that do not.
- Updated `mock-services.yml` to streamline mock configurations, ensuring compatibility with the new testing workflows.
- Added `requires-api-keys` tags to relevant test cases across various test files, improving organization and clarity regarding API dependencies.
- Enhanced documentation for test scripts and configurations, providing clearer guidance for contributors on executing tests based on API key requirements.
* Add optional service profile to Docker Compose test configuration
* Refactor audio processing and job handling for transcription workflows
- Updated `upload_and_process_audio_files` and `start_post_conversation_jobs` to enqueue transcription jobs separately for file uploads, ensuring accurate processing order.
- Enhanced logging to provide clearer insights into job enqueuing and processing stages.
- Removed batch transcription from the post-conversation job chain for streaming audio, utilizing the streaming transcript directly.
- Introduced word-level timestamps in the `Conversation` model to improve transcript detail and accuracy.
- Updated tests to reflect changes in job handling and ensure proper verification of post-conversation processing.
* Remove unnecessary network aliases from speaker service in Docker Compose configuration
* Add network aliases for speaker service in Docker Compose configuration
* Refactor Conversation model to use string for provider field
- Updated the `Conversation` model to replace the `TranscriptProvider` enum with a string type for the `provider` field, allowing for greater flexibility in provider names.
- Adjusted related job functions to accommodate this change, simplifying provider handling in the transcription workflow.
* Enhance configuration and model handling for waveform data
- Updated Docker Compose files to mount the entire config directory, allowing for better management of configuration files.
- Introduced a new `WaveformData` model to store pre-computed waveform visualization data, improving UI performance by enabling waveform display without real-time decoding.
- Enhanced the `app_factory` and `job` models to include the new `WaveformData` model, ensuring proper initialization and data handling.
- Implemented waveform generation logic in a new worker module, allowing for on-demand waveform creation from audio chunks.
- Added API endpoints for retrieving and generating waveform data, improving the overall audio processing capabilities.
- Updated tests to cover new functionality and ensure robustness in waveform data handling.
* Add SDK testing scripts for authentication, conversation retrieval, and audio upload
- Introduced three new test scripts: `sdk_test_auth.py`, `sdk_test_conversations.py`, and `sdk_test_upload.py`.
- Each script tests different functionalities of the SDK, including authentication, conversation retrieval, and audio file uploads.
- The scripts utilize the `ChronicleClient` to perform operations and print results for verification.
- Enhanced testing capabilities for the SDK, ensuring robust validation of core features.
* Enhance audio processing and conversation handling for large files
- Added configuration options for speaker recognition chunking in `.env.template`, allowing for better management of large audio files.
- Updated `get_conversations` function to include an `include_deleted` parameter for filtering conversations based on their deletion status.
- Enhanced `finalize_session` method in `AudioStreamProducer` to send an end marker to Redis, ensuring proper session closure.
- Introduced `reconstruct_audio_segments` function to yield audio segments with overlap for efficient processing of lengthy conversations.
- Implemented merging of overlapping speaker segments to improve accuracy in speaker recognition.
- Added integration tests for WebSocket streaming transcription to validate the end_marker functionality and overall transcription flow.
* archive
* Implement annotation system and enhance audio processing capabilities
- Introduced a new annotation model to support user edits and AI-powered suggestions for memories and transcripts.
- Added annotation routes for CRUD operations, enabling the creation and management of annotations via the API.
- Enhanced the audio processing workflow to support fetching audio segments from the backend, improving speaker recognition accuracy.
- Updated the speaker recognition client to handle conversation-based audio fetching, allowing for better management of large audio files.
- Implemented a cron job for generating AI suggestions on potential errors in transcripts and memories, improving user experience and content accuracy.
- Enhanced the web UI to support inline editing of transcript segments and memory content, providing a more interactive user experience.
- Updated configuration files to support new features and improve overall system flexibility.
* Implement OmegaConf-based configuration management for backend settings
- Introduced a new configuration loader using OmegaConf for unified management of backend settings.
- Updated existing configuration functions to leverage the new loader, enhancing flexibility and maintainability.
- Added support for environment variable interpolation in configuration files.
- Refactored various components to retrieve settings from the new configuration system, improving consistency across the application.
- Updated requirements to include OmegaConf as a dependency.
- Enhanced documentation and comments for clarity on configuration management.
* Refactor .env.template and remove unused diarization configuration
- Updated the .env.template to clarify its purpose for secret values and streamline setup instructions.
- Removed the deprecated diarization_config.json.template file, as it is no longer needed.
- Added new environment variables for Langfuse and Tailscale integration to enhance observability and remote service access.
* Implement legacy environment variable syntax support in configuration loader
- Added custom OmegaConf resolvers to handle legacy ${VAR:-default} syntax for backward compatibility.
- Introduced a preprocessing function to convert legacy syntax in YAML files to OmegaConf-compatible format.
- Updated the load_config function to utilize the new preprocessing for loading defaults and user configurations.
- Enhanced documentation for clarity on the new legacy syntax handling.
* Add plugins configuration path retrieval and refactor usage
- Introduced a new function `get_plugins_yml_path` to centralize the retrieval of the plugins.yml file path.
- Updated `system_controller.py` and `plugin_service.py` to use the new function for improved maintainability and consistency in accessing the plugins configuration.
- Enhanced code clarity by removing hardcoded paths and utilizing the centralized configuration method.
* Unify plugin terminology and fix memory job dependencies
Plugin terminology: subscriptions→events, trigger→condition
Memory jobs: no longer blocked by disabled speaker recognition
* Update Docker Compose configuration and enhance system routes
- Updated Docker Compose files to mount the entire config directory, consolidating configuration management.
- Refactored the `save_diarization_settings` function to improve clarity and maintainability by renaming it to `save_diarization_settings_controller`.
- Enhanced the System component in the web UI to include configuration diagnostics, providing better visibility into system health and issues.
* circular import
* Refactor testing infrastructure and enhance container management
- Updated the testing documentation to reflect a new Makefile-based approach for running tests and managing containers.
- Introduced new scripts for container management, including starting, stopping, restarting, and cleaning containers while preserving logs.
- Added a cleanup script to handle data ownership and permissions correctly.
- Implemented a logging system that saves container logs automatically before cleanup.
- Enhanced the README with detailed instructions for running tests and managing the test environment.
* Add Email Summarizer Plugin and SMTP Email Service
- Introduced the Email Summarizer Plugin that automatically sends email summaries upon conversation completion.
- Implemented SMTP Email Service for sending emails, supporting HTML and plain text formats with TLS/SSL encryption.
- Added configuration options for SMTP settings in the .env.template and plugins.yml.template.
- Created comprehensive documentation for plugin development and usage, including a new plugin generation script.
- Enhanced testing coverage for the Email Summarizer Plugin and SMTP Email Service to ensure reliability and functionality.
* Refactor plugin management and introduce Email Summarizer setup
- Removed the static PLUGINS dictionary and replaced it with a dynamic discovery mechanism for plugins.
- Implemented a new setup process for plugins, allowing for configuration via individual setup scripts.
- Added the Email Summarizer plugin with a dedicated setup script for SMTP configuration.
- Enhanced the main setup flow to support community plugins and their configuration.
- Cleaned up unused functions related to plugin configuration and streamlined the overall plugin setup process.
* Enhance plugin configuration and documentation
- Updated the .env.template to include new configuration options for the Home Assistant and Email Summarizer plugins, including server URLs, tokens, and additional settings.
- Refactored Docker Compose files to correctly mount plugin configuration paths.
- Introduced comprehensive documentation for plugin configuration architecture, detailing the separation of concerns for orchestration, settings, and secrets.
- Added individual configuration files for the Home Assistant and Email Summarizer plugins, ensuring proper management of non-secret settings and environment variable references.
- Improved the plugin loading process to merge configurations from multiple sources, enhancing flexibility and maintainability.
* Refactor plugin setup process to allow interactive user input
- Updated the plugin setup script to run interactively, enabling plugins to prompt for user input during configuration.
- Removed output capturing to facilitate real-time interaction and improved error messaging to include exit codes for better debugging.
* Add shared setup utilities for interactive configuration
- Introduced `setup_utils.py` containing functions for reading environment variables, prompting user input, and masking sensitive values.
- Refactored existing code in `wizard.py` and `init.py` to utilize these shared utilities, improving code reuse and maintainability.
- Updated documentation to include usage examples for the new utilities in plugin setup scripts, enhancing developer experience and clarity.
* Enhance plugin security architecture and configuration management
- Introduced a three-file separation for plugin configuration to improve security:
- `backends/advanced/.env` for secrets (gitignored)
- `config/plugins.yml` for orchestration with environment variable references
- `plugins/{plugin_id}/config.yml` for non-secret defaults
- Updated documentation to emphasize the importance of using `${ENV_VAR}` syntax for sensitive data and provided examples of correct usage.
- Enhanced the Email Summarizer plugin setup process to automatically update `config/plugins.yml` with environment variable references, ensuring secrets are not hardcoded.
- Added new fields to the User model for notification email management and improved error logging in user-related functions.
- Refactored audio chunk utilities to use a consistent method for fetching conversation metadata.
* Refactor backend components for improved functionality and stability
- Added a new parameter `transcript_version_id` to the `open_conversation_job` function to support streaming transcript versioning.
- Enhanced error handling in `check_enrolled_speakers_job` and `recognise_speakers_job` to allow conversations to proceed even when the speaker service is unavailable, improving resilience.
- Updated `send_to_adv.py` to support dynamic WebSocket and HTTP protocols based on environment settings, enhancing configuration flexibility.
- Introduced a background task in `send_to_adv.py` to handle incoming messages from the backend, ensuring connection stability and logging interim results.
* Refactor plugin setup timing to enhance configuration flow
* Refactor save_diarization_settings_controller to improve validation and error handling
- Updated the controller to filter out invalid settings instead of raising an error for each unknown key, allowing for more flexible input.
- Added a check to reject requests with no valid settings provided, enhancing robustness.
- Adjusted logging to reflect the filtered settings being saved.
* Refactor audio processing and conversation management for improved deduplication and tracking
* Refactor audio and email handling for improved functionality and security
- Updated `mask_value` function to handle whitespace more effectively.
- Enhanced `create_plugin` to remove existing directories when using the `--force` option.
- Changed logging level from error to debug for existing admin user checks.
- Improved client ID generation logging for clarity.
- Removed unused fields from conversation creation.
- Added HTML escaping in email templates to prevent XSS attacks.
- Updated audio file download function to include user ID for better tracking.
- Adjusted WebSocket connection settings to respect SSL verification based on environment variables.
* Refactor audio upload functionality to remove unused parameters
- Removed `auto_generate_client` and `folder` parameters from audio upload functions to streamline the API.
- Updated related function calls and documentation to reflect these changes, enhancing clarity and reducing complexity.
* Refactor Email Summarizer plugin configuration for improved clarity and security
- Removed outdated migration instructions from `plugin-configuration.md` to streamline documentation.
- Enhanced `README.md` to clearly outline the three-file separation for plugin configuration, emphasizing the roles of `.env`, `config.yml`, and `plugins.yml`.
- Updated `setup.py` to reflect changes in orchestration settings, ensuring only relevant configurations are included in `config/plugins.yml`.
- Improved security messaging to highlight the importance of not committing secrets to version control.
* Update API key configuration in config.yml.template to use environment variable syntax for improved flexibility and security. This change standardizes the way API keys are referenced across different models and services. (#273)
Co-authored-by: roshan.john <roshanjohn1460@gmail.com>
* Refactor Redis job queue cleanup process for improved success tracking
- Replaced total job count with separate counters for successful and failed jobs during Redis queue cleanup.
- Enhanced logging to provide detailed feedback on the number of jobs cleared and any failures encountered.
- Improved error handling to ensure job counts are accurately reflected even when exceptions occur.
* fix tests
* Update CI workflows to use 'docker compose' for log retrieval and added container status check
- Replaced 'docker logs' commands with 'docker compose -f docker-compose-test.yml logs' for consistency across workflows.
- Added a check for running containers before saving logs to enhance debugging capabilities.
* test fixes
* FIX StreamingTranscriptionConsumer to support cumulative audio timestamp adjustments
- Added `audio_offset_seconds` to track cumulative audio duration for accurate timestamp adjustments across transcription sessions.
- Updated `store_final_result` method to adjust word and segment timestamps based on cumulative audio offset.
- Improved logging to reflect changes in audio offset after storing results.
- Modified Makefile and documentation to clarify test execution options, including new tags for slow and SDK tests, enhancing test organization and execution clarity.
* Enhance test container setup and improve error messages in integration tests
- Set `COMPOSE_PROJECT_NAME` for test containers to ensure consistent naming.
- Consolidated error messages in the `websocket_transcription_e2e_test.robot` file for clarity, improving readability and debugging.
* Improve WebSocket closing logic and enhance integration test teardown
- Added timeout handling for WebSocket closure in `AudioStreamClient` to prevent hanging and ensure clean disconnection.
- Updated integration tests to log the total chunks sent when closing audio streams, improving clarity on resource management during test teardown.
* Refactor job status handling to align with RQ standards
- Updated job status checks across various modules to use "started" and "finished" instead of "processing" and "completed" for consistency with RQ's naming conventions.
- Adjusted related logging and response messages to reflect the new status terminology.
- Simplified Docker Compose project name handling in test scripts to avoid conflicts and improve clarity in test environment setup.
* Update test configurations and improve audio inactivity handling
- Increased `SPEECH_INACTIVITY_THRESHOLD_SECONDS` to 20 seconds in `docker-compose-test.yml` for better audio duration handling during tests.
- Refactored session handling in `session_controller.py` to clarify client ID usage.
- Updated `conversation_utils.py` to track speech activity using audio timestamps, enhancing accuracy in inactivity detection.
- Simplified test scripts by removing unnecessary `COMPOSE_PROJECT_NAME` references, aligning with the new project naming convention.
- Adjusted integration tests to reflect changes in inactivity timeout and ensure proper handling of audio timestamps.
* Refactor audio processing and enhance error handling
- Updated `worker_orchestrator.py` to use `logger.exception` for improved error logging.
- Changed default MongoDB database name from "friend-lite" to "chronicle" in multiple files for consistency.
- Added a new method `close_stream_without_stop` in `audio_stream_client.py` to handle abrupt WebSocket disconnections.
- Enhanced audio validation in `audio_utils.py` to support automatic resampling of audio data if sample rates do not match.
- Improved logging in various modules to provide clearer insights during audio processing and event dispatching.
* Enhance Docker command handling and configuration management
- Updated `run_compose_command` to support separate build commands for services, including profile management for backend and speaker-recognition services.
- Improved error handling and output streaming during Docker command execution.
- Added `ensure_docker_network` function to verify and create the required Docker network before starting services.
- Refactored configuration files to utilize `oc.env` for environment variable management, ensuring better compatibility and flexibility across different environments.
* Enhance configuration loading to support custom config file paths
- Added support for the CONFIG_FILE environment variable to allow specifying custom configuration files for testing.
- Implemented logic to handle both absolute paths and relative filenames for the configuration file, improving flexibility in configuration management.
* Update test scripts to use TEST_CONFIG_FILE for configuration management
- Replaced CONFIG_FILE with TEST_CONFIG_FILE in both run-no-api-tests.sh and run-robot-tests.sh to standardize configuration file usage.
- Updated paths to point to mock and deepgram-openai configuration files inside the container, improving clarity and consistency in test setups.
* Refactor audio upload response handling and improve error reporting
- Updated `upload_and_process_audio_files` to return appropriate HTTP status codes based on upload results: 400 for all failures, 207 for partial successes, and 200 for complete success.
- Enhanced error messages in the audio upload tests to provide clearer feedback on upload failures, including specific error details for better debugging.
- Adjusted test scripts to ensure consistent handling of conversation IDs in job metadata, improving validation checks for job creation.
* Refactor audio processing and job handling to improve transcription management
- Updated `upload_and_process_audio_files` to check for transcription provider availability before enqueueing jobs, enhancing error handling and logging.
- Modified `start_post_conversation_jobs` to conditionally enqueue memory extraction jobs based on configuration, improving flexibility in job management.
- Enhanced event dispatch job dependencies to only include jobs that were actually enqueued, ensuring accurate job tracking.
- Added `is_transcription_available` function to check transcription provider status, improving modularity and clarity in the transcription workflow.
* Enhance integration tests for plugin events and improve error handling
- Updated integration tests to filter plugin events by conversation ID, ensuring accurate event tracking and reducing noise from fixture events.
- Improved error messages in event verification to include conversation ID context, enhancing clarity during test failures.
- Refactored audio upload handling to check for transcription job creation, allowing for more robust conversation polling and error reporting.
- Added new keyword to verify conversation end reasons, improving test coverage for conversation state validation.
* Enhance speaker recognition testing and audio processing
- Added mock speaker recognition client to facilitate testing without resource-intensive dependencies.
- Updated Docker Compose configurations to include mock speaker client for test environments.
- Refactored audio segment reconstruction to ensure precise clipping based on time boundaries.
- Improved error handling in transcription jobs and speaker recognition workflows to enhance robustness.
- Adjusted integration tests to utilize real-time pacing for audio chunk streaming, improving test accuracy.
* Refactor audio chunk retrieval and enhance logging in audio processing
- Introduced logging for audio chunk requests to improve traceability.
- Replaced manual audio chunk processing with a dedicated `reconstruct_audio_segment` function for better clarity and efficiency.
- Improved error handling during audio reconstruction to provide more informative responses in case of failures.
- Cleaned up imports and removed redundant code related to audio chunk calculations.
* Refactor mock speaker recognition client and improve testing structure
- Replaced direct import of mock client with a structured import from the new testing module.
- Introduced a dedicated `mock_speaker_client.py` to provide a mock implementation for speaker recognition, facilitating testing without heavy dependencies.
- Added an `__init__.py` file in the testing directory to organize testing utilities and mocks.
* Enhance conversation model to include word-level timestamps and improve transcript handling
- Added a new `words` field to the `Conversation` model for storing word-level timestamps.
- Updated methods to handle word data during transcript version creation, ensuring compatibility with speaker recognition.
- Refactored conversation job processing to utilize the new word structure, improving data integrity and access.
- Enhanced speaker recognition job to read words from the new standardized location, ensuring backward compatibility with legacy data.
* Implement speaker reprocessing feature and enhance timeout calculation
- Added a new endpoint to reprocess speaker identification for existing transcripts, creating a new version with re-identified speakers.
- Introduced a method…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The edit UI looks like this
Summary by CodeRabbit
New Features
Tests
Chores
✏️ Tip: You can customize this high-level summary in your review settings.