Enhance configuration management and add new setup scripts#235
Merged
AnkushMalaker merged 12 commits intodevfrom Jan 2, 2026
Merged
Enhance configuration management and add new setup scripts#235AnkushMalaker merged 12 commits intodevfrom
AnkushMalaker merged 12 commits intodevfrom
Conversation
- Updated .gitignore to include config.yml and its template. - Added config.yml.template for default configuration settings. - Introduced restart.sh script for service management. - Enhanced services.py to load config.yml and check for Obsidian/Neo4j integration. - Updated wizard.py to prompt for Obsidian/Neo4j configuration during setup and create config.yml from template if it doesn't exist.
Contributor
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the 📝 WalkthroughWalkthroughThis PR centralizes configuration management by moving Changes
Sequence Diagram(s)sequenceDiagram
participant Wizard as Setup Wizard<br/>(wizard.py)
participant ConfigMgr as ConfigManager<br/>(config_manager.py)
participant Init as Init Script<br/>(init.py)
participant Registry as Model Registry<br/>(model_registry.py)
participant Services as Backend Services
Wizard->>ConfigMgr: Load config/config.yml
ConfigMgr-->>Wizard: config data
Wizard->>Wizard: Prompt for providers<br/>(STT, LLM, Memory)
Wizard->>ConfigMgr: Update defaults & models
ConfigMgr->>ConfigMgr: Save config.yml<br/>+ .env backup
ConfigMgr-->>Wizard: success
Wizard->>Init: Invoke init.py with<br/>CLI flags
Init->>ConfigMgr: Load/validate config
ConfigMgr-->>Init: config data
Init->>Init: Setup optional services<br/>(Obsidian, STT, LLM)
Init->>ConfigMgr: Update config & .env
ConfigMgr-->>Init: persisted
Services->>Registry: get_models_registry()
Registry-->>Services: Models with defaults
Services->>ConfigMgr: Load memory/speaker<br/>config
ConfigMgr-->>Services: provider settings
Services->>Services: Initialize providers<br/>from registry config
Estimated code review effort🎯 4 (Complex) | ⏱️ ~65 minutes Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- Updated Docker Compose files to include the new Neo4j service configuration. - Added support for Obsidian/Neo4j integration in the setup process. - Refactored transcription providers to utilize a registry-driven approach for Deepgram and Parakeet. - Enhanced error handling and logging in transcription processes. - Improved environment variable management in test scripts to prioritize command-line overrides. - Removed deprecated Parakeet provider implementation and streamlined audio stream workers.
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Actions performedReview triggered.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (2)
backends/advanced/init.py (2)
637-641: Conditional will never be true.This check for
OBSIDIAN_ENABLED == 'true'will never succeed becausesetup_obsidian()doesn't set this config key. See the fix proposed above.
657-665: Same issue - OBSIDIAN_ENABLED not set.This conditional for showing the
--profile obsidiancommand suffers from the same issue.
🧹 Nitpick comments (9)
backends/advanced/src/advanced_omi_backend/services/transcription/__init__.py (1)
129-148: Uselogger.debug()instead oflogger.info()for debug messages.The messages are prefixed with "DEBUG" but use
logger.info(). Debug-level logging should uselogger.debug()to allow proper log level filtering in production environments.🔎 Proposed fix
- # DEBUG: Log Deepgram response structure if "results" in data and "channels" in data.get("results", {}): channels = data["results"]["channels"] if channels and "alternatives" in channels[0]: alt = channels[0]["alternatives"][0] - logger.info(f"DEBUG Registry: Deepgram alternative keys: {list(alt.keys())}") + logger.debug(f"Registry: Deepgram alternative keys: {list(alt.keys())}") - # DEBUG: Log what we extracted - logger.info(f"DEBUG Registry: Extracted {len(segments)} segments from response") + logger.debug(f"Registry: Extracted {len(segments)} segments from response") if segments and len(segments) > 0: - logger.info(f"DEBUG Registry: First segment keys: {list(segments[0].keys()) if isinstance(segments[0], dict) else 'not a dict'}") - logger.info(f"DEBUG Registry: First segment: {segments[0]}") + logger.debug(f"Registry: First segment keys: {list(segments[0].keys()) if isinstance(segments[0], dict) else 'not a dict'}") + logger.debug(f"Registry: First segment: {segments[0]}")services.py (1)
18-29: Consider usinglogging.exception()for better error diagnostics.The broad exception catch is acceptable for a config-loading utility that gracefully falls back. However, using
logging.exception()instead of printing would provide stack traces for debugging config parsing issues.Based on learnings, prefer
logging.exception()in except blocks to automatically capture stack traces.🔎 Proposed refinement
+import logging + +logger = logging.getLogger(__name__) + def load_config_yml(): """Load config.yml from repository root""" config_path = Path(__file__).parent / 'config.yml' if not config_path.exists(): return None try: with open(config_path, 'r') as f: return yaml.safe_load(f) except Exception as e: - console.print(f"[yellow]⚠️ Warning: Could not load config.yml: {e}[/yellow]") + logger.exception("Could not load config.yml") + console.print(f"[yellow]⚠️ Warning: Could not load config.yml: {e}[/yellow]") return Nonebackends/advanced/run-test.sh (1)
42-52: Override logic has inconsistent condition.Line 51 checks if
DEEPGRAM_API_KEYis set AND_TRANSCRIPTION_PROVIDER_OVERRIDEis empty to determine whether to use CI/environment variables. This condition seems unrelated - the presence ofDEEPGRAM_API_KEYdoesn't necessarily indicate CI environment usage whenTRANSCRIPTION_PROVIDERwasn't overridden.Consider simplifying to check for any pre-existing override variable, or document the intent more clearly:
-if [ -n "$DEEPGRAM_API_KEY" ] && [ -z "$_TRANSCRIPTION_PROVIDER_OVERRIDE" ]; then +if [ -n "$_DEEPGRAM_API_KEY_OVERRIDE" ] || [ -n "$_TRANSCRIPTION_PROVIDER_OVERRIDE" ]; then print_info "Using environment variables from CI/environment..."backends/advanced/docker-compose.yml (2)
180-201: Neo4j service configuration looks good with minor notes.The service is well-configured for the Obsidian integration with profile-based activation. A few observations:
Default password:
${NEO4J_PASSWORD:-password}- the default is weak but acceptable for local development.APOC unrestricted:
NEO4J_dbms_security_procedures_unrestricted=apoc.*grants full access to APOC procedures. This is fine for development but should be restricted in production deployments.Volume mismatch: The service uses bind mounts (
./data/neo4j_data:/data) but there are also named volumes declared at lines 235-238 (neo4j_data,neo4j_logs). The named volumes are unused - consider removing them or switching to named volumes for consistency withmongo_data.
235-238: Unused volume definitions.These named volumes (
neo4j_data,neo4j_logs) are declared but not used - theneo4j-mem0service uses bind mounts (./data/neo4j_data:/data) instead. Consider removing these unused declarations to avoid confusion.🔎 Proposed fix
volumes: ollama_data: driver: local mongo_data: driver: local caddy_data: driver: local caddy_config: driver: local - neo4j_data: - driver: local - neo4j_logs: - driver: localwizard.py (3)
419-432: Fix f-string without placeholder and consider improving password UX.Line 429 has an f-string without any placeholders. Also, consider showing the default password value in the confirmation message for clarity.
🔎 Proposed fix
except EOFError: neo4j_password = "neo4jpassword" - console.print(f"Using default password") + console.print("Using default password: neo4jpassword") break
450-461: Variable shadowing causes confusion.Line 451 resets
obsidian_enabled = False, shadowing the value from the earlier prompt (line 409). While this post-run validation by reading.envis defensive, it's confusing. Consider using a different variable name likeobsidian_configured:🔎 Proposed fix
- # Check for Obsidian/Neo4j configuration - obsidian_enabled = False + # Verify Obsidian/Neo4j was configured successfully + obsidian_configured = False if 'advanced' in selected_services and 'advanced' not in failed_services: backend_env_path = Path('backends/advanced/.env') if backend_env_path.exists(): neo4j_host = read_env_value(str(backend_env_path), 'NEO4J_HOST') obsidian_enabled_flag = read_env_value(str(backend_env_path), 'OBSIDIAN_ENABLED') if neo4j_host and not is_placeholder(neo4j_host, 'your-neo4j-host-here', 'your_neo4j_host_here'): - obsidian_enabled = True + obsidian_configured = True elif obsidian_enabled_flag == 'true': - obsidian_enabled = True + obsidian_configured = TrueAnd update line 470:
- if obsidian_enabled: + if obsidian_configured:
463-473: Remove extraneous f-string prefixes.Lines 463 and 471 have f-strings without placeholders, as flagged by static analysis.
🔎 Proposed fix
# Final Summary - console.print(f"\n🎊 [bold green]Setup Complete![/bold green]") + console.print("\n🎊 [bold green]Setup Complete![/bold green]") console.print(f"✅ {success_count}/{len(selected_services)} services configured successfully")# Inform about Obsidian/Neo4j if configured if obsidian_enabled: - console.print(f"\n📚 [bold cyan]Obsidian Integration Detected[/bold cyan]") + console.print("\n📚 [bold cyan]Obsidian Integration Detected[/bold cyan]")backends/advanced/src/advanced_omi_backend/services/transcription/deepgram.py (1)
78-80: Chain exception when re-raising.Based on project learnings, when re-raising exceptions, use
raise ... from eto preserve the original context for better debuggability.🔎 Proposed fix
except Exception as e: logger.error(f"Deepgram transcription failed: {e}", exc_info=True) - raise + raise RuntimeError(f"Transcription failed: {e}") from eOr if you want to preserve the original exception type:
except Exception as e: logger.error(f"Deepgram transcription failed: {e}", exc_info=True) - raise + raise e from e
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (16)
.gitignorebackends/advanced/docker-compose-test.ymlbackends/advanced/docker-compose.ymlbackends/advanced/init.pybackends/advanced/run-test.shbackends/advanced/src/advanced_omi_backend/services/transcription/__init__.pybackends/advanced/src/advanced_omi_backend/services/transcription/deepgram.pybackends/advanced/src/advanced_omi_backend/services/transcription/parakeet.pybackends/advanced/src/advanced_omi_backend/services/transcription/parakeet_stream_consumer.pybackends/advanced/src/advanced_omi_backend/workers/audio_stream_deepgram_worker.pybackends/advanced/src/advanced_omi_backend/workers/audio_stream_parakeet_worker.pybackends/advanced/src/advanced_omi_backend/workers/memory_jobs.pyconfig.yml.templaterestart.shservices.pywizard.py
💤 Files with no reviewable changes (1)
- backends/advanced/src/advanced_omi_backend/services/transcription/parakeet.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.py: Use Black formatter with 100-character line length for Python code
Use isort for Python import organization
ALL imports must be at the top of the file after the docstring - never import modules in the middle of functions or files
Group imports in Python files: standard library, third-party, then local imports
Use lazy imports sparingly and only when absolutely necessary for circular import issues in Python
Always raise errors in Python, never silently ignore - use explicit error handling with proper exceptions rather than silent failures
Avoid defensivehasattr()checks in Python - research and understand input/response or class structure instead
Files:
backends/advanced/src/advanced_omi_backend/services/transcription/__init__.pyservices.pybackends/advanced/src/advanced_omi_backend/services/transcription/parakeet_stream_consumer.pywizard.pybackends/advanced/src/advanced_omi_backend/workers/memory_jobs.pybackends/advanced/src/advanced_omi_backend/services/transcription/deepgram.pybackends/advanced/src/advanced_omi_backend/workers/audio_stream_parakeet_worker.pybackends/advanced/src/advanced_omi_backend/workers/audio_stream_deepgram_worker.pybackends/advanced/init.py
🧠 Learnings (3)
📚 Learning: 2025-12-08T23:52:34.959Z
Learnt from: AnkushMalaker
Repo: chronicler-ai/chronicle PR: 178
File: backends/advanced/src/advanced_omi_backend/services/memory/providers/mycelia.py:218-223
Timestamp: 2025-12-08T23:52:34.959Z
Learning: In Python code (chronicle project), prefer logging.exception() inside except blocks to automatically log the full stack trace. When re-raising exceptions, always chain with 'raise ... from e' to preserve the original context; use 'raise ... from None' only if you explicitly want to suppress the context. This improves debuggability across Python files.
Applied to files:
backends/advanced/src/advanced_omi_backend/services/transcription/__init__.pyservices.pybackends/advanced/src/advanced_omi_backend/services/transcription/parakeet_stream_consumer.pywizard.pybackends/advanced/src/advanced_omi_backend/workers/memory_jobs.pybackends/advanced/src/advanced_omi_backend/services/transcription/deepgram.pybackends/advanced/src/advanced_omi_backend/workers/audio_stream_parakeet_worker.pybackends/advanced/src/advanced_omi_backend/workers/audio_stream_deepgram_worker.pybackends/advanced/init.py
📚 Learning: 2025-12-20T01:11:51.495Z
Learnt from: CR
Repo: chronicler-ai/chronicle PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-20T01:11:51.495Z
Learning: Use `uv run python3` instead of direct `python3` commands for Python execution in the project
Applied to files:
restart.sh
📚 Learning: 2025-12-20T01:11:51.495Z
Learnt from: CR
Repo: chronicler-ai/chronicle PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-20T01:11:51.495Z
Learning: Use `uv` for Python dependency management in backend projects instead of pip
Applied to files:
restart.sh
🧬 Code graph analysis (7)
backends/advanced/src/advanced_omi_backend/services/transcription/__init__.py (1)
backends/advanced/src/advanced_omi_backend/models/conversation.py (1)
segments(197-199)
services.py (1)
backends/advanced/init.py (1)
load_config_yml(129-143)
backends/advanced/src/advanced_omi_backend/services/transcription/parakeet_stream_consumer.py (1)
backends/advanced/src/advanced_omi_backend/services/transcription/__init__.py (1)
get_transcription_provider(257-280)
backends/advanced/run-test.sh (4)
backends/advanced/setup-https.sh (2)
print_info(15-17)print_warning(23-25)tests/run-robot-tests.sh (2)
print_info(16-18)print_warning(24-26)run-test.sh (2)
print_info(17-19)print_warning(25-27)extras/speaker-recognition/run-test.sh (2)
print_info(34-36)print_warning(42-44)
backends/advanced/src/advanced_omi_backend/workers/memory_jobs.py (1)
backends/advanced/src/advanced_omi_backend/models/conversation.py (1)
transcript(191-193)
backends/advanced/src/advanced_omi_backend/services/transcription/deepgram.py (4)
backends/advanced/src/advanced_omi_backend/services/audio_stream/consumer.py (2)
BaseAudioStreamConsumer(19-591)transcribe_audio(244-257)backends/advanced/src/advanced_omi_backend/services/transcription/__init__.py (2)
get_transcription_provider(257-280)transcribe(71-150)backends/advanced/src/advanced_omi_backend/services/transcription/base.py (5)
mode(71-73)mode(88-89)mode(122-123)transcribe(46-61)transcribe(126-134)backends/advanced/src/advanced_omi_backend/services/transcription/parakeet_stream_consumer.py (2)
_ConcreteConsumer(45-78)transcribe_audio(50-78)
backends/advanced/init.py (1)
extras/speaker-recognition/init.py (1)
prompt_password(56-66)
🪛 Ruff (0.14.10)
services.py
27-27: Do not catch blind exception: Exception
(BLE001)
113-113: String contains ambiguous ℹ (INFORMATION SOURCE). Did you mean i (LATIN SMALL LETTER I)?
(RUF001)
backends/advanced/src/advanced_omi_backend/services/transcription/parakeet_stream_consumer.py
40-42: Avoid specifying long messages outside the exception class
(TRY003)
wizard.py
329-329: String contains ambiguous ℹ (INFORMATION SOURCE). Did you mean i (LATIN SMALL LETTER I)?
(RUF001)
423-423: Possible hardcoded password assigned to: "neo4j_password"
(S105)
428-428: Possible hardcoded password assigned to: "neo4j_password"
(S105)
429-429: f-string without any placeholders
Remove extraneous f prefix
(F541)
463-463: f-string without any placeholders
Remove extraneous f prefix
(F541)
471-471: f-string without any placeholders
Remove extraneous f prefix
(F541)
backends/advanced/src/advanced_omi_backend/services/transcription/deepgram.py
41-43: Avoid specifying long messages outside the exception class
(TRY003)
🔇 Additional comments (19)
backends/advanced/src/advanced_omi_backend/workers/memory_jobs.py (1)
92-97: LGTM! Good fallback handling for short segment content.The fallback logic appropriately handles cases where speaker recognition fails or segments contain insufficient text, ensuring that valid transcript content isn't discarded. The compound condition safely validates the fallback source, and the subsequent length check on line 99 ensures the conversation is still processable after the fallback.
backends/advanced/docker-compose-test.yml (1)
35-35: LGTM!The environment variable is correctly propagated to both services, maintaining consistency with the config-driven transcription provider approach.
Also applies to: 147-147
.gitignore (1)
7-8: LGTM!The ignore pattern correctly excludes user-specific
config.ymlwhile keeping the template tracked, following best practices for configuration management.backends/advanced/src/advanced_omi_backend/services/transcription/parakeet_stream_consumer.py (1)
26-42: LGTM!The refactor to registry-based provider loading is well-implemented with appropriate error handling. The error message clearly guides users to check their
config.ymlconfiguration.config.yml.template (3)
131-131: LGTM!The change from boolean
falseto string'true'is correct. Deepgram's API expects string values ('true'/'false') for query parameters, not booleans.
194-194: LGTM!Good cleanup removing the placeholder text from the extraction prompt.
205-208: LGTM!The Obsidian configuration block correctly supports the new Obsidian/Neo4j integration feature with appropriate default values.
restart.sh (1)
1-2: LGTM!The restart script correctly uses
uv runper project standards and properly invokes the services management script.backends/advanced/src/advanced_omi_backend/workers/audio_stream_parakeet_worker.py (1)
30-36: LGTM!The updates correctly reflect the shift to config-driven transcription providers and provide helpful guidance to users about
config.ymlconfiguration.Also applies to: 51-54
services.py (1)
92-113: LGTM!The Obsidian/Neo4j enablement detection correctly prioritizes
config.ymlover.envfallback, implementing a clean configuration hierarchy. The logic properly appends the obsidian profile when needed.backends/advanced/src/advanced_omi_backend/workers/audio_stream_deepgram_worker.py (2)
30-36: Behavior change when API key is missing.The worker now logs warnings and continues rather than exiting early when
DEEPGRAM_API_KEYis not set. However,DeepgramStreamConsumernow usesget_transcription_provider(mode="batch")which will raiseRuntimeErrorif config.yml doesn't have a defaultsttmodel configured (seedeepgram.pylines 40-43).This means the worker will fail at consumer initialization rather than at startup validation. Consider whether earlier validation might provide a better user experience.
51-55: LGTM!The comment accurately reflects the new registry-driven approach, and the consumer instantiation correctly omits the removed
api_keyparameter.backends/advanced/run-test.sh (2)
70-92: LGTM!The override restoration logic correctly gives command-line arguments highest priority. Appropriately, sensitive API keys (DEEPGRAM, OPENAI) are restored without printing their values.
196-215: LGTM!The conditional cleanup logic is a useful addition for debugging test failures. The implementation consistently applies to both failure and success paths, and the default behavior (cleanup enabled) is preserved.
backends/advanced/init.py (2)
711-711: LGTM!The
setup_obsidian()call is correctly placed in the setup flow aftersetup_optional_services().
754-759: LGTM!The CLI arguments are correctly defined. Note that passing
--neo4j-passwordvia command line may expose the password in process listings, but this is acceptable for interactive setup scripts.wizard.py (1)
153-172: LGTM!The function signature extension and command construction correctly propagate Obsidian configuration to the backend setup script.
backends/advanced/src/advanced_omi_backend/services/transcription/deepgram.py (2)
46-76: LGTM!The inner
_ConcreteConsumerclass correctly implements thetranscribe_audiomethod following the same pattern asparakeet_stream_consumer.py. The confidence calculation logic is sound.
24-34: LGTM!The constructor properly validates that a transcription provider is available before proceeding, providing a clear error message if config.yml is misconfigured.
Comment on lines
418
to
462
| def setup_obsidian(self): | ||
| """Configure Obsidian/Neo4j integration""" | ||
| # Check if enabled via command line | ||
| if hasattr(self.args, 'enable_obsidian') and self.args.enable_obsidian: | ||
| enable_obsidian = True | ||
| neo4j_password = getattr(self.args, 'neo4j_password', None) | ||
|
|
||
| if not neo4j_password: | ||
| self.console.print("[yellow][WARNING][/yellow] --enable-obsidian provided but no password") | ||
| neo4j_password = self.prompt_password("Neo4j password (min 8 chars)") | ||
| else: | ||
| # Interactive prompt (fallback) | ||
| self.console.print() | ||
| self.console.print("[bold cyan]Obsidian/Neo4j Integration[/bold cyan]") | ||
| self.console.print("Enable graph-based knowledge management for Obsidian vault notes") | ||
| self.console.print() | ||
|
|
||
| try: | ||
| enable_obsidian = Confirm.ask("Enable Obsidian/Neo4j integration?", default=False) | ||
| except EOFError: | ||
| self.console.print("Using default: No") | ||
| enable_obsidian = False | ||
|
|
||
| if enable_obsidian: | ||
| neo4j_password = self.prompt_password("Neo4j password (min 8 chars)") | ||
|
|
||
| if enable_obsidian: | ||
| # Update .env with credentials | ||
| self.config["NEO4J_HOST"] = "neo4j-mem0" | ||
| self.config["NEO4J_USER"] = "neo4j" | ||
| self.config["NEO4J_PASSWORD"] = neo4j_password | ||
|
|
||
| # Update config.yml with feature flag | ||
| if "memory" not in self.config_yml_data: | ||
| self.config_yml_data["memory"] = {} | ||
| if "obsidian" not in self.config_yml_data["memory"]: | ||
| self.config_yml_data["memory"]["obsidian"] = {} | ||
|
|
||
| self.config_yml_data["memory"]["obsidian"]["enabled"] = True | ||
| self.config_yml_data["memory"]["obsidian"]["neo4j_host"] = "neo4j-mem0" | ||
| self.config_yml_data["memory"]["obsidian"]["timeout"] = 30 | ||
|
|
||
| self.console.print("[green][SUCCESS][/green] Obsidian/Neo4j configured") | ||
| self.console.print("[blue][INFO][/blue] Neo4j will start automatically with --profile obsidian") | ||
|
|
Contributor
There was a problem hiding this comment.
Missing OBSIDIAN_ENABLED assignment in config.
The setup_obsidian method sets NEO4J_HOST, NEO4J_USER, and NEO4J_PASSWORD in self.config, but never sets OBSIDIAN_ENABLED. However, show_summary() (line 638) and show_next_steps() (line 660) check for self.config.get('OBSIDIAN_ENABLED') == 'true', which will always be False.
🔎 Proposed fix
if enable_obsidian:
# Update .env with credentials
+ self.config["OBSIDIAN_ENABLED"] = "true"
self.config["NEO4J_HOST"] = "neo4j-mem0"
self.config["NEO4J_USER"] = "neo4j"
self.config["NEO4J_PASSWORD"] = neo4j_password🤖 Prompt for AI Agents
In backends/advanced/init.py around lines 418-462, when enable_obsidian is True
the code updates NEO4J_* entries in self.config but never sets OBSIDIAN_ENABLED;
add an assignment self.config["OBSIDIAN_ENABLED"] = "true" (and optionally set
to "false" when not enabled) alongside the NEO4J entries so show_summary() and
show_next_steps() which check self.config.get('OBSIDIAN_ENABLED') == 'true'
behave correctly.
Comment on lines
35
to
43
| from advanced_omi_backend.services.audio_stream.consumer import BaseAudioStreamConsumer | ||
| from advanced_omi_backend.services.transcription import get_transcription_provider | ||
|
|
||
| self.api_key = api_key or os.getenv("DEEPGRAM_API_KEY") | ||
| if not self.api_key: | ||
| raise ValueError("DEEPGRAM_API_KEY is required") | ||
|
|
||
| # Initialize Deepgram provider | ||
| self.provider = DeepgramProvider(api_key=self.api_key) | ||
| # Get registry-driven transcription provider | ||
| self.provider = get_transcription_provider(mode="batch") | ||
| if not self.provider: | ||
| raise RuntimeError( | ||
| "Failed to load transcription provider. Ensure config.yml has a default 'stt' model configured." | ||
| ) |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check if moving imports to top would cause circular import issues
echo "=== Checking import dependencies ==="
echo "--- get_transcription_provider location ---"
rg -l "def get_transcription_provider" --type=py
echo "--- What does transcription/__init__.py import? ---"
find . -path "*/services/transcription/__init__.py" -type f -exec cat {} \;
echo "--- What imports deepgram.py? ---"
rg -l "from.*deepgram import|import.*deepgram" --type=py -g '!**/test*'
echo "--- Full deepgram.py file structure ---"
find . -name "deepgram.py" -path "*/transcription/*" -type f -exec wc -l {} \; -exec head -60 {} \;
echo "--- Check if BaseAudioStreamConsumer imports from transcription ---"
rg -A 10 "class BaseAudioStreamConsumer" --type=pyRepository: chronicler-ai/chronicle
Length of output: 16184
Move imports to top of file after docstring.
Lines 35-36 import BaseAudioStreamConsumer and get_transcription_provider inside the constructor, violating the coding guideline that all imports must be at the top of the file. There are no circular import issues in the import chain—transcription/__init__.py does not import from deepgram.py or audio_stream.consumer, making lazy imports unnecessary. Move these imports to the module level after the docstring.
🧰 Tools
🪛 Ruff (0.14.10)
41-43: Avoid specifying long messages outside the exception class
(TRY003)
🤖 Prompt for AI Agents
In backends/advanced/src/advanced_omi_backend/services/transcription/deepgram.py
around lines 35 to 43, two imports (BaseAudioStreamConsumer and
get_transcription_provider) are inside the constructor; move these imports to
the top of the module immediately after the file docstring so all imports are
module-level, then remove the local imports from the constructor and use the
top-level names directly; no other logic changes are required since there are no
circular import issues.
…matrix (#237) * Update configuration management and enhance file structure - Refactored configuration file paths to use a dedicated `config/` directory, including updates to `config.yml` and its template. - Modified service scripts to load the new configuration path for `config.yml`. - Enhanced `.gitignore` to include the new configuration files and templates. - Updated documentation to reflect changes in configuration file locations and usage. - Improved setup scripts to ensure proper creation and management of configuration files. - Added new test configurations for various provider combinations to streamline testing processes. * Add test requirements and clean up imports in wizard.py - Introduced a new `test-requirements.txt` file to manage testing dependencies. - Removed redundant import of `shutil` in `wizard.py` to improve code clarity. * Add ConfigManager for unified configuration management - Introduced a new `config_manager.py` module to handle reading and writing configurations from `config.yml` and `.env` files, ensuring backward compatibility. - Refactored `ChronicleSetup` in `backends/advanced/init.py` to utilize `ConfigManager` for loading and updating configurations, simplifying the setup process. - Removed redundant methods for loading and saving `config.yml` directly in `ChronicleSetup`, as these are now managed by `ConfigManager`. - Enhanced user feedback during configuration updates, including success messages for changes made to configuration files. * Refactor transcription provider configuration and enhance setup process - Updated `.env.template` to clarify speech-to-text configuration and removed deprecated options for Mistral. - Modified `docker-compose.yml` to streamline environment variable management by removing unused Mistral keys. - Enhanced `ChronicleSetup` in `init.py` to provide clearer user feedback and updated the transcription provider selection process to rely on `config.yml`. - Improved error handling in the websocket controller to determine the transcription provider from the model registry instead of environment variables. - Updated health check routes to reflect the new method of retrieving the transcription provider from `config.yml`. - Adjusted `config.yml.template` to include comments on transcription provider options for better user guidance. * Enhance ConfigManager with deep merge functionality - Updated the `update_memory_config` method to perform a deep merge of updates into the memory configuration, ensuring nested dictionaries are merged correctly. - Added a new `_deep_merge` method to handle recursive merging of dictionaries, improving configuration management capabilities. * Refactor run-test.sh and enhance memory extraction tests - Removed deprecated environment variable handling for TRANSCRIPTION_PROVIDER in `run-test.sh`, streamlining the configuration process. - Introduced a new `run-custom.sh` script for executing Robot tests with custom configurations, improving test flexibility. - Enhanced memory extraction tests in `audio_keywords.robot` and `memory_keywords.robot` to include detailed assertions and result handling. - Updated `queue_keywords.robot` to fail fast if a job is in a 'failed' state when expecting 'completed', improving error handling. - Refactored `test_env.py` to load environment variables with correct precedence, ensuring better configuration management. * unify tests to robot test, add some more clean up * Update health check configuration in docker-compose-test.yml (#241) - Increased the number of retries from 5 to 10 for improved resilience during service readiness checks. - Extended the start period from 30s to 60s to allow more time for services to initialize before health checks commence. * Add step to create test configuration file in robot-tests.yml - Introduced a new step in the GitHub Actions workflow to copy the test configuration file from tests/configs/deepgram-openai.yml to a new config/config.yml. - Added logging to confirm the creation of the test config file, improving visibility during the test setup process. * remove cache step since not required * coderabbit comments * Refactor ConfigManager error handling for configuration file loading - Updated the ConfigManager to raise RuntimeError exceptions when the configuration file is not found or is invalid, improving error visibility and user guidance. - Removed fallback behavior that previously returned the current directory, ensuring users are explicitly informed about missing or invalid configuration files. * Refactor _find_repo_root method in ConfigManager - Updated the _find_repo_root method to locate the repository root using the __file__ location instead of searching for config/config.yml, simplifying the logic and improving reliability. - Removed the previous error handling that raised a RuntimeError if the configuration file was not found, as the new approach assumes config_manager.py is always at the repo root.
|
| Metric | Count |
|---|---|
| ✅ Passed | 91 |
| ❌ Failed | 1 |
| 📊 Total | 92 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
|
| Metric | Count |
|---|---|
| ✅ Passed | 91 |
| ❌ Failed | 1 |
| 📊 Total | 92 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
) * Enhance speaker recognition service integration and error handling - Updated `docker-compose-test.yml` to enable speaker recognition in the test environment and added a new `speaker-service-test` service for testing purposes. - Refactored `run-test.sh` to improve the execution of Robot Framework tests from the repository root. - Enhanced error handling in `speaker_recognition_client.py` to return detailed error messages for connection issues. - Improved error logging in `speaker_jobs.py` to handle and report errors from the speaker recognition service more effectively. - Updated `Dockerfile` to copy the full source code after dependencies are cached, ensuring all necessary files are included in the image. * Remove integration tests workflow and enhance robot tests with HF_TOKEN verification - Deleted the `integration-tests.yml` workflow file to streamline CI processes. - Updated `robot-tests.yml` to include verification for the new `HF_TOKEN` secret, ensuring all required secrets are checked before running tests. * Fix key access in system admin tests to use string indexing for speakers data * Refactor Robot Framework tests and enhance error handling in memory services - Removed the creation of the test environment file from the GitHub Actions workflow to streamline setup. - Updated the Robot Framework tests to utilize a unified test script for improved consistency. - Enhanced error messages in the MemoryService class to provide more context on connection failures for LLM and vector store providers. - Added critical checks for API key presence in the OpenAIProvider class to ensure valid credentials are provided before proceeding. - Adjusted various test setup scripts to use a centralized BACKEND_DIR variable for better maintainability and clarity. * Refactor test container cleanup in run-robot-tests.sh - Updated the script to dynamically construct container names from docker-compose services, improving maintainability and reducing hardcoded values. - Enhanced the cleanup process for stuck test containers by utilizing the COMPOSE_PROJECT_NAME variable.
|
| Metric | Count |
|---|---|
| ✅ Passed | 92 |
| ❌ Failed | 2 |
| 📊 Total | 94 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
|
| Metric | Count |
|---|---|
| ✅ Passed | 92 |
| ❌ Failed | 2 |
| 📊 Total | 94 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
- Set absolute paths for consistent directory references to simplify navigation. - Capture container logs, status, and resource usage for better debugging. - Refactor cleanup process to utilize dynamic backend directory references, improving maintainability. - Ensure proper navigation back to the tests directory after operations.
|
| Metric | Count |
|---|---|
| ✅ Passed | 91 |
| ❌ Failed | 3 |
| 📊 Total | 94 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
- Introduced speaker recognition settings in config.yml.template, allowing for easy enable/disable and service URL configuration. - Updated run-robot-tests.sh to use a test-specific configuration file that disables speaker recognition for improved CI performance. - Modified deepgram-openai.yml to disable speaker recognition during CI tests to enhance execution speed.
Collaborator
Author
|
Who knew running ML models in github CI is slow as hell? |
|
| Metric | Count |
|---|---|
| ✅ Passed | 92 |
| ❌ Failed | 2 |
| 📊 Total | 94 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html - HTML reports
- robot-test-results-xml - XML output
- Updated docker-compose-test.yml to clarify speaker recognition settings, now controlled via config.yml for improved CI performance. - Enhanced model_registry.py to include a dedicated speaker_recognition field for better configuration handling. - Modified speaker_recognition_client.py to load configuration from config.yml, allowing for dynamic enabling/disabling of the speaker recognition service based on the configuration.
🎉 Robot Framework Test ResultsStatus: ✅ All tests passed!
📊 View ReportsGitHub Pages (Live Reports): Download Artifacts:
|
Collaborator
Author
|
@coderabbitai review, can you also see about resolving old comments you left on this PR |
- Introduced a new keyword to verify that the minimum number of workers are registered, enhancing the robustness of health checks. - Updated the worker count validation test to include a wait mechanism for worker registration, improving test reliability. - Clarified comments regarding expected worker counts to reflect the distinction between RQ and audio stream workers.
Contributor
🧩 Analysis chain@AnkushMalaker, I'll trigger a review and check on my previous comments to see which ones can be resolved. 🏁 Script executed: Length of output: 7016 🏁 Script executed: Length of output: 1472 In wizard.py around lines 316 to 330, the function setup_config_file() performs 🏁 Script executed: Length of output: 220 🏁 Script executed: Length of output: 881 🏁 Script executed: Length of output: 575 🏁 Script executed: Length of output: 150 🏁 Script executed: Length of output: 1723 @AnkushMalaker, I've checked my previous comments on this PR. Here's the status: Review Comments Status✅ Already Resolved (1 comment)
I'll trigger a fresh review now. ✅ Actions performedReview triggered.
|
0xrushi
approved these changes
Jan 2, 2026
Contributor
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (5)
backends/advanced/src/advanced_omi_backend/workers/speaker_jobs.py (3)
40-41: Move imports to module level.Per coding guidelines, all imports must be at the top of the file. These imports for
TranscriptionResultsAggregatorandSpeakerRecognitionClientshould be moved to the module-level imports section.🔎 Proposed fix
Add at the top of the file (after line 14):
from advanced_omi_backend.services.audio_stream import TranscriptionResultsAggregator from advanced_omi_backend.speaker_recognition_client import SpeakerRecognitionClientThen remove lines 40-41 from the function body.
94-95: Moverqimport to module level.Per coding guidelines, imports should be at the top of the file.
rqis a third-party library and unlikely to cause circular import issues.🔎 Proposed fix
Add at the top of the file (in the third-party imports section):
from rq import get_current_jobThen remove line 94 from the function body and change line 95 to:
- from rq import get_current_job - current_job = get_current_job() + current_job = get_current_job()
149-150: Move imports to module level.These imports should be at the top of the file per coding guidelines. Note that
SpeakerRecognitionClientis also imported incheck_enrolled_speakers_job(line 41) - consolidating at module level eliminates duplication.🔎 Proposed fix
Add at the top of the file:
from advanced_omi_backend.models.conversation import Conversation
SpeakerRecognitionClientis already suggested to be moved from line 41.backends/advanced/src/advanced_omi_backend/services/memory/providers/llm_providers.py (1)
60-68: Move imports to module level per coding guidelines.The conditional imports on lines 62 and 66 violate the project guideline: "ALL imports must be at the top of the file after the docstring - never import modules in the middle of functions or files." The dynamic type wrapper on line 67 is also unnecessarily complex.
Both imports can be safely moved to the top of the file, and the runtime selection can happen without the dynamic wrapper.
As per coding guidelines, lazy imports should only be used for circular import issues, which don't apply here.
🔎 Proposed refactor
Move imports to the top of the file (after line 16):
from typing import Any, Dict, List, Optional +import langfuse.openai as langfuse_openai +from openai import AsyncOpenAI, OpenAI + from ..base import LLMProviderBaseThen simplify the function:
def _get_openai_client(api_key: str, base_url: str, is_async: bool = False): """Get OpenAI client with optional Langfuse tracing. Args: api_key: OpenAI API key base_url: OpenAI API base URL is_async: Whether to return async or sync client Returns: OpenAI client instance (with or without Langfuse tracing) """ if _is_langfuse_enabled(): - # Use Langfuse-wrapped OpenAI for tracing - import langfuse.openai as openai memory_logger.debug("Using OpenAI client with Langfuse tracing") + if is_async: + return langfuse_openai.AsyncOpenAI(api_key=api_key, base_url=base_url) + else: + return langfuse_openai.OpenAI(api_key=api_key, base_url=base_url) else: - # Use regular OpenAI client without tracing - from openai import AsyncOpenAI, OpenAI - openai = type('OpenAI', (), {'OpenAI': OpenAI, 'AsyncOpenAI': AsyncOpenAI})() memory_logger.debug("Using OpenAI client without tracing") - - if is_async: - return openai.AsyncOpenAI(api_key=api_key, base_url=base_url) - else: - return openai.OpenAI(api_key=api_key, base_url=base_url) + if is_async: + return AsyncOpenAI(api_key=api_key, base_url=base_url) + else: + return OpenAI(api_key=api_key, base_url=base_url)backends/advanced/Docs/quickstart.md (1)
559-559: Potential typo: "gpt-5-mini" should likely be "gpt-4o-mini".Line 559 references "gpt-5-mini" but other locations in this file (lines 142, 609, 665) correctly use "gpt-4o-mini". This appears to be an inconsistency.
🔎 Suggested fix
-OPENAI_MODEL=gpt-4o-mini # RECOMMENDED: "gpt-5-mini" for better memory extraction +OPENAI_MODEL=gpt-4o-mini # RECOMMENDED: "gpt-4o-mini" for better memory extraction
♻️ Duplicate comments (1)
backends/advanced/init.py (1)
407-424: MissingOBSIDIAN_ENABLEDassignment in self.config.The
show_summary()(line 607) andshow_next_steps()(line 629) methods checkself.config.get('OBSIDIAN_ENABLED') == 'true', but this value is never set when enabling Obsidian. OnlyNEO4J_*values are assigned.🔎 Proposed fix
if enable_obsidian: # Update .env with credentials + self.config["OBSIDIAN_ENABLED"] = "true" self.config["NEO4J_HOST"] = "neo4j-mem0" self.config["NEO4J_USER"] = "neo4j" self.config["NEO4J_PASSWORD"] = neo4j_password
🧹 Nitpick comments (18)
backends/advanced/src/advanced_omi_backend/workers/speaker_jobs.py (1)
325-336: Uselogger.exception()for automatic stack trace logging.Based on project learnings, prefer
logging.exception()inside except blocks to automatically log the full stack trace at ERROR level. This also eliminates the need for the inlinetracebackimport.🔎 Proposed fix
except Exception as speaker_error: - logger.error(f"❌ Speaker recognition failed: {speaker_error}") - import traceback - logger.debug(traceback.format_exc()) + logger.exception(f"❌ Speaker recognition failed: {speaker_error}") return { "success": False,tests/endpoints/system_admin_tests.robot (1)
54-54: Consider consistent quote style for dictionary keys.Line 54 uses single quotes (
'primary_speakers') while line 82 uses double quotes ("speakers"). Both are valid Python, but PEP 8 recommends consistency within a codebase.🔎 Proposed refactor for consistency
- Should Be True isinstance($config['primary_speakers'], list) + Should Be True isinstance($config["primary_speakers"], list)Also applies to: 82-82
tests/infrastructure/infra_tests.robot (1)
47-53: Consider inlining verification or clarify guideline exception for retry scenarios.This keyword abstracts verification logic, which conflicts with the guideline: "Write verifications directly in Robot Framework tests inline, not abstracted to keywords."
However, it's used with
Wait Until Keyword Succeeds(line 189-190) which requires a keyword for retry logic. Consider either:
- Clarifying that retry scenarios are an acceptable exception to the inline verification guideline
- Restructuring to separate data retrieval from verification (though this requires calling the endpoint twice)
Based on coding guidelines.
backends/advanced/src/advanced_omi_backend/services/memory/providers/llm_providers.py (1)
191-203: Excellent fail-fast validation for API credentials.The early validation prevents cryptic downstream failures when API keys are missing. The descriptive error messages guide users to fix configuration issues quickly.
Note: Static analysis suggests moving long messages into custom exception classes (TRY003), but this is optional for configuration validation like this.
backends/advanced/run-test.sh (2)
101-141: Solid config-driven provider selection with appropriate fallback.The migration from environment variable to config.yml as the source of truth for STT provider selection is well-executed. The fallback to
TRANSCRIPTION_PROVIDERwith a clear warning ensures backward compatibility during the transition.Optional: Consider explicit error reporting for config reading failures
The current silent error handling (
2>/dev/null || echo "") suppresses all failures when reading from config.yml. While the fallback with warning is appropriate for backward compatibility, you could improve debuggability by capturing and logging specific errors:-STT_PROVIDER=$(uv run python -c " +STT_PROVIDER_OUTPUT=$(uv run python -c " from advanced_omi_backend.model_registry import get_models_registry registry = get_models_registry() if registry and registry.defaults: stt_model = registry.get_default('stt') if stt_model: print(stt_model.model_provider or '') -" 2>/dev/null || echo "") +" 2>&1) + +if [ $? -eq 0 ]; then + STT_PROVIDER=$STT_PROVIDER_OUTPUT +else + STT_PROVIDER="" + print_warning "Failed to read STT provider from config.yml: $STT_PROVIDER_OUTPUT" +fiThis would help users identify import errors, missing config files, or invalid YAML without changing the fallback behavior.
221-240: Consider extracting duplicated cleanup logic.The conditional cleanup logic (lines 222-228 and 234-240) is duplicated. While functionally correct, extracting it into a function would improve maintainability and reduce the risk of inconsistent updates.
Optional: Extract cleanup logic into a function
+cleanup_containers() { + if [ "${CLEANUP_CONTAINERS:-true}" != "false" ]; then + print_info "Cleaning up test containers..." + docker compose -f docker-compose-test.yml down -v || true + docker system prune -f || true + else + print_warning "Skipping cleanup (CLEANUP_CONTAINERS=false) - containers left running" + fi +} + # Run the Robot Framework integration tests with extended timeout (mem0 needs time for comprehensive extraction) # IMPORTANT: Robot tests must be run from the repository root where backends/ and tests/ are siblings print_info "Starting Robot Framework integration tests (timeout: 15 minutes)..." if (cd ../.. && timeout 900 robot --outputdir test-results --loglevel INFO tests/integration/integration_test.robot); then print_success "Integration tests completed successfully!" else TEST_EXIT_CODE=$? print_error "Integration tests FAILED with exit code: $TEST_EXIT_CODE" - # Clean up test containers before exiting (unless CLEANUP_CONTAINERS=false) - if [ "${CLEANUP_CONTAINERS:-true}" != "false" ]; then - print_info "Cleaning up test containers after failure..." - docker compose -f docker-compose-test.yml down -v || true - docker system prune -f || true - else - print_warning "Skipping cleanup (CLEANUP_CONTAINERS=false) - containers left running for debugging" - fi + cleanup_containers exit $TEST_EXIT_CODE fi -# Clean up test containers (unless CLEANUP_CONTAINERS=false) -if [ "${CLEANUP_CONTAINERS:-true}" != "false" ]; then - print_info "Cleaning up test containers..." - docker compose -f docker-compose-test.yml down -v || true - docker system prune -f || true -else - print_warning "Skipping cleanup (CLEANUP_CONTAINERS=false) - containers left running" -fi +cleanup_containerstests/test-requirements.txt (1)
6-6: Pin the websockets version to ensure test stability.The
websocketsdependency is currently unpinned. Version 15.0 introduced backwards-incompatible changes (including automatic proxy handling and threading keepalive behavior), so pinning to the current stable version (15.0.1) will prevent unexpected test failures from future updates.tests/run-custom.sh (1)
18-20: Relative path may break if script is run from a different directory.The
CONFIG_FILEpath assumes the script is executed from a specific directory (likelytests/). Consider using$(dirname "$0")to make the path relative to the script's location.🔎 Proposed fix
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" + -CONFIG_FILE="../tests/configs/${CONFIG_NAME}.yml" \ +CONFIG_FILE="${SCRIPT_DIR}/configs/${CONFIG_NAME}.yml" \ PARAKEET_ASR_URL="${PARAKEET_URL}" \ - ./run-robot-tests.sh + "${SCRIPT_DIR}/run-robot-tests.sh"tests/configs/parakeet-ollama.yml (1)
17-18: Consider using a dedicated env var for Ollama API key.Using
${OPENAI_API_KEY:-ollama}works because Ollama's OpenAI-compatible API accepts any non-empty key, but it may cause confusion. A dedicatedOLLAMA_API_KEYwithollamadefault would be clearer.🔎 Suggested improvement
- api_key: ${OPENAI_API_KEY:-ollama} + api_key: ${OLLAMA_API_KEY:-ollama}Also applies to: 30-31
backends/advanced/src/advanced_omi_backend/speaker_recognition_client.py (2)
168-179: Uselogging.exceptionfor automatic stack traces.In except blocks,
logging.exception()automatically includes the full traceback. Based on learnings, this improves debuggability.🔎 Proposed fix
except ClientConnectorError as e: - logger.error(f"🎤 Failed to connect to speaker recognition service: {e}") + logger.exception(f"🎤 Failed to connect to speaker recognition service: {e}") return {"error": "connection_failed", "message": str(e), "segments": []} except asyncio.TimeoutError as e: - logger.error(f"🎤 Timeout connecting to speaker recognition service: {e}") + logger.exception(f"🎤 Timeout connecting to speaker recognition service: {e}") return {"error": "timeout", "message": str(e), "segments": []} except aiohttp.ClientError as e: logger.warning(f"🎤 Client error during speaker recognition: {e}") return {"error": "client_error", "message": str(e), "segments": []} except Exception as e: - logger.error(f"🎤 Error during speaker recognition: {e}") + logger.exception(f"🎤 Error during speaker recognition: {e}") return {"error": "unknown_error", "message": str(e), "segments": []}
289-302: Same logging improvement needed here.Apply
logging.exceptionin the except blocks fordiarize_and_identifyas well, per project guidelines.🔎 Proposed fix
except ClientConnectorError as e: - logger.error(f"🎤 [DIARIZE] ❌ Failed to connect to speaker recognition service at {self.service_url}: {e}") + logger.exception(f"🎤 [DIARIZE] ❌ Failed to connect to speaker recognition service at {self.service_url}: {e}") return {"error": "connection_failed", "message": str(e), "segments": []} except asyncio.TimeoutError as e: - logger.error(f"🎤 [DIARIZE] ❌ Timeout connecting to speaker recognition service: {e}") + logger.exception(f"🎤 [DIARIZE] ❌ Timeout connecting to speaker recognition service: {e}") return {"error": "timeout", "message": str(e), "segments": []} except aiohttp.ClientError as e: logger.warning(f"🎤 [DIARIZE] ❌ Client error during speaker recognition: {e}") return {"error": "client_error", "message": str(e), "segments": []} except Exception as e: - logger.error(f"🎤 [DIARIZE] ❌ Error during speaker diarization and identification: {e}") + logger.exception(f"🎤 [DIARIZE] ❌ Error during speaker diarization and identification: {e}") import traceback logger.debug(traceback.format_exc()) return {"error": "unknown_error", "message": str(e), "segments": []}wizard.py (2)
431-431: Remove extraneous f-prefix.This string has no placeholders.
🔎 Proposed fix
- console.print(f"Using default password") + console.print("Using default password")
465-465: Remove extraneous f-prefix.This string has no placeholders.
🔎 Proposed fix
- console.print(f"\n🎊 [bold green]Setup Complete![/bold green]") + console.print("\n🎊 [bold green]Setup Complete![/bold green]")config_manager.py (4)
107-114: Chain exceptions withraise ... from efor better debugging.Per project guidelines, re-raised exceptions should preserve the original context using
from e.🔎 Proposed fix
except yaml.YAMLError as e: raise RuntimeError( f"Invalid YAML in configuration file {self.config_yml_path}: {e}" - ) + ) from e except Exception as e: raise RuntimeError( f"Failed to load configuration file {self.config_yml_path}: {e}" - ) + ) from e
132-134: Uselogging.exceptionfor automatic stack traces.Based on learnings,
logging.exception()should be used in except blocks to automatically include the traceback.🔎 Proposed fix
except Exception as e: - logger.error(f"Failed to save config.yml: {e}") + logger.exception(f"Failed to save config.yml: {e}") raise
180-182: Same logging improvement needed here.🔎 Proposed fix
except Exception as e: - logger.error(f"Failed to update .env file: {e}") + logger.exception(f"Failed to update .env file: {e}") raise
362-367: Singleton may unexpectedly replace cached instance.When
service_pathis provided, the function always creates a new instance, overwriting_config_manager. This could lead to unexpected behavior if different callers pass differentservice_pathvalues. Consider documenting this behavior or adding a warning when replacing an existing instance.🔎 Proposed fix (add logging for awareness)
def get_config_manager(service_path: Optional[str] = None) -> ConfigManager: global _config_manager if _config_manager is None or service_path is not None: + if _config_manager is not None and service_path is not None: + logger.debug(f"Replacing existing ConfigManager (service_path={service_path})") _config_manager = ConfigManager(service_path=service_path) return _config_managerbackends/advanced/init.py (1)
698-698: Remove extraneous f-prefix.This string has no placeholders.
🔎 Proposed fix
- self.console.print(f" • ../../config/config.yml - Model and memory provider configuration") + self.console.print(" • ../../config/config.yml - Model and memory provider configuration")
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (54)
.github/workflows/README.md.github/workflows/integration-tests.yml.github/workflows/robot-tests.yml.gitignoreCLAUDE.mdDocs/getting-started.mdbackends/advanced/.env.templatebackends/advanced/Docs/README.mdbackends/advanced/Docs/contribution.mdbackends/advanced/Docs/memories.mdbackends/advanced/Docs/memory-configuration-guide.mdbackends/advanced/Docs/quickstart.mdbackends/advanced/README.mdbackends/advanced/SETUP_SCRIPTS.mdbackends/advanced/docker-compose-test.ymlbackends/advanced/docker-compose.ymlbackends/advanced/init.pybackends/advanced/run-test.shbackends/advanced/src/advanced_omi_backend/controllers/websocket_controller.pybackends/advanced/src/advanced_omi_backend/model_registry.pybackends/advanced/src/advanced_omi_backend/routers/modules/health_routes.pybackends/advanced/src/advanced_omi_backend/services/memory/providers/chronicle.pybackends/advanced/src/advanced_omi_backend/services/memory/providers/llm_providers.pybackends/advanced/src/advanced_omi_backend/speaker_recognition_client.pybackends/advanced/src/advanced_omi_backend/workers/speaker_jobs.pybackends/advanced/start-workers.shbackends/advanced/tests/test_integration.pyconfig/README.mdconfig/config.yml.templateconfig_manager.pyextras/speaker-recognition/run-test.shservices.pytests/Makefiletests/configs/README.mdtests/configs/deepgram-openai.ymltests/configs/full-local.ymltests/configs/parakeet-ollama.ymltests/configs/parakeet-openai.ymltests/endpoints/rq_queue_tests.robottests/endpoints/system_admin_tests.robottests/infrastructure/infra_tests.robottests/integration/integration_test.robottests/resources/audio_keywords.robottests/resources/memory_keywords.robottests/resources/queue_keywords.robottests/run-custom.shtests/run-robot-tests.shtests/setup/setup_keywords.robottests/setup/teardown_keywords.robottests/setup/test_data.pytests/setup/test_env.pytests/setup/test_manager_keywords.robottests/test-requirements.txtwizard.py
💤 Files with no reviewable changes (2)
- .github/workflows/integration-tests.yml
- backends/advanced/tests/test_integration.py
✅ Files skipped from review due to trivial changes (6)
- backends/advanced/Docs/memories.md
- tests/configs/README.md
- backends/advanced/Docs/contribution.md
- tests/configs/deepgram-openai.yml
- config/README.md
- tests/configs/full-local.yml
🚧 Files skipped from review as they are similar to previous changes (1)
- .gitignore
🧰 Additional context used
📓 Path-based instructions (2)
**/*.robot
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.robot: Only use the 11 approved tags from @tests/tags.md in Robot Framework tests - custom tags are not permitted
Use tab-separated tags in Robot Framework tests (e.g.,[Tags] infra audio-streaming), never space-separated
Write verifications directly in Robot Framework tests inline, not abstracted to keywords
Follow the Arrange-Act-Assert pattern in Robot Framework tests with inline verifications
Use descriptive Robot Framework test and keyword names that explain business purpose, not technical implementation
Files:
tests/resources/queue_keywords.robottests/resources/audio_keywords.robottests/endpoints/system_admin_tests.robottests/endpoints/rq_queue_tests.robottests/resources/memory_keywords.robottests/setup/test_manager_keywords.robottests/integration/integration_test.robottests/setup/teardown_keywords.robottests/infrastructure/infra_tests.robottests/setup/setup_keywords.robot
**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.py: Use Black formatter with 100-character line length for Python code
Use isort for Python import organization
ALL imports must be at the top of the file after the docstring - never import modules in the middle of functions or files
Group imports in Python files: standard library, third-party, then local imports
Use lazy imports sparingly and only when absolutely necessary for circular import issues in Python
Always raise errors in Python, never silently ignore - use explicit error handling with proper exceptions rather than silent failures
Avoid defensivehasattr()checks in Python - research and understand input/response or class structure instead
Files:
backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.pybackends/advanced/src/advanced_omi_backend/routers/modules/health_routes.pyconfig_manager.pybackends/advanced/src/advanced_omi_backend/model_registry.pywizard.pytests/setup/test_data.pybackends/advanced/src/advanced_omi_backend/services/memory/providers/chronicle.pybackends/advanced/src/advanced_omi_backend/speaker_recognition_client.pybackends/advanced/src/advanced_omi_backend/workers/speaker_jobs.pybackends/advanced/init.pybackends/advanced/src/advanced_omi_backend/services/memory/providers/llm_providers.pytests/setup/test_env.pyservices.py
🧠 Learnings (9)
📚 Learning: 2025-12-20T01:11:51.495Z
Learnt from: CR
Repo: chronicler-ai/chronicle PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-20T01:11:51.495Z
Learning: Applies to **/*.robot : Use descriptive Robot Framework test and keyword names that explain business purpose, not technical implementation
Applied to files:
tests/resources/queue_keywords.robot.github/workflows/README.mdtests/resources/audio_keywords.robotCLAUDE.mdtests/resources/memory_keywords.robottests/run-robot-tests.shtests/setup/test_manager_keywords.robottests/integration/integration_test.robottests/setup/teardown_keywords.robottests/infrastructure/infra_tests.robottests/setup/setup_keywords.robot
📚 Learning: 2025-12-20T01:11:51.495Z
Learnt from: CR
Repo: chronicler-ai/chronicle PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-20T01:11:51.495Z
Learning: Before writing any Robot Framework test, read [tests/TESTING_GUIDELINES.md](tests/TESTING_GUIDELINES.md) for comprehensive testing patterns and standards
Applied to files:
.github/workflows/README.md.github/workflows/robot-tests.ymltests/MakefileCLAUDE.mdtests/run-robot-tests.sh
📚 Learning: 2025-12-20T01:11:51.495Z
Learnt from: CR
Repo: chronicler-ai/chronicle PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-20T01:11:51.495Z
Learning: Use `uv` for Python dependency management in backend projects instead of pip
Applied to files:
.github/workflows/robot-tests.yml
📚 Learning: 2025-12-20T01:11:51.495Z
Learnt from: CR
Repo: chronicler-ai/chronicle PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-20T01:11:51.495Z
Learning: Use `uv run python3` instead of direct `python3` commands for Python execution in the project
Applied to files:
.github/workflows/robot-tests.yml
📚 Learning: 2025-12-20T01:11:51.495Z
Learnt from: CR
Repo: chronicler-ai/chronicle PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-20T01:11:51.495Z
Learning: Applies to **/*.robot : Write verifications directly in Robot Framework tests inline, not abstracted to keywords
Applied to files:
tests/Makefiletests/integration/integration_test.robottests/infrastructure/infra_tests.robot
📚 Learning: 2025-12-08T23:52:34.959Z
Learnt from: AnkushMalaker
Repo: chronicler-ai/chronicle PR: 178
File: backends/advanced/src/advanced_omi_backend/services/memory/providers/mycelia.py:218-223
Timestamp: 2025-12-08T23:52:34.959Z
Learning: In Python code (chronicle project), prefer logging.exception() inside except blocks to automatically log the full stack trace. When re-raising exceptions, always chain with 'raise ... from e' to preserve the original context; use 'raise ... from None' only if you explicitly want to suppress the context. This improves debuggability across Python files.
Applied to files:
backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.pybackends/advanced/src/advanced_omi_backend/routers/modules/health_routes.pyconfig_manager.pybackends/advanced/src/advanced_omi_backend/model_registry.pywizard.pytests/setup/test_data.pybackends/advanced/src/advanced_omi_backend/services/memory/providers/chronicle.pybackends/advanced/src/advanced_omi_backend/speaker_recognition_client.pybackends/advanced/src/advanced_omi_backend/workers/speaker_jobs.pybackends/advanced/init.pybackends/advanced/src/advanced_omi_backend/services/memory/providers/llm_providers.pytests/setup/test_env.pyservices.py
📚 Learning: 2025-12-20T01:11:51.495Z
Learnt from: CR
Repo: chronicler-ai/chronicle PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-20T01:11:51.495Z
Learning: Applies to **/*.robot : Use tab-separated tags in Robot Framework tests (e.g., `[Tags] infra audio-streaming`), never space-separated
Applied to files:
tests/run-robot-tests.sh
📚 Learning: 2025-12-20T01:11:51.495Z
Learnt from: CR
Repo: chronicler-ai/chronicle PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-20T01:11:51.495Z
Learning: Before writing Robot Framework test code, scan relevant resource files for existing keywords - never write code that duplicates existing keywords
Applied to files:
tests/integration/integration_test.robot
📚 Learning: 2025-12-20T01:11:51.495Z
Learnt from: CR
Repo: chronicler-ai/chronicle PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-20T01:11:51.495Z
Learning: Check if src/ is volume mounted in Docker - if not, run `docker compose build` so code changes are reflected (do not simply run `docker compose restart`)
Applied to files:
tests/setup/setup_keywords.robot
🧬 Code graph analysis (4)
backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.py (1)
backends/advanced/src/advanced_omi_backend/model_registry.py (2)
get_models_registry(342-357)get_default(200-222)
backends/advanced/src/advanced_omi_backend/routers/modules/health_routes.py (3)
backends/advanced/src/advanced_omi_backend/model_registry.py (1)
get_default(200-222)backends/advanced/src/advanced_omi_backend/services/transcription/__init__.py (2)
name(68-69)name(167-168)backends/advanced/src/advanced_omi_backend/services/transcription/base.py (1)
name(65-67)
backends/advanced/src/advanced_omi_backend/speaker_recognition_client.py (1)
backends/advanced/src/advanced_omi_backend/model_registry.py (1)
get_models_registry(342-357)
backends/advanced/init.py (1)
config_manager.py (4)
ConfigManager(37-344)get_full_config(328-335)update_config_defaults(310-326)update_memory_config(263-282)
🪛 LanguageTool
backends/advanced/Docs/README.md
[style] ~91-~91: You have already used this phrasing in nearby sentences. Consider replacing it to add variety to your writing.
Context: ...yml` - Configuration overview ### "I want to work on memory extraction" 1. [memori...
(REP_WANT_TO_VB)
🪛 Ruff (0.14.10)
backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.py
310-310: Avoid specifying long messages outside the exception class
(TRY003)
314-314: Avoid specifying long messages outside the exception class
(TRY003)
318-318: Avoid specifying long messages outside the exception class
(TRY003)
config_manager.py
93-96: Avoid specifying long messages outside the exception class
(TRY003)
102-105: Abstract raise to an inner function
(TRY301)
102-105: Avoid specifying long messages outside the exception class
(TRY003)
108-110: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
108-110: Avoid specifying long messages outside the exception class
(TRY003)
111-111: Do not catch blind exception: Exception
(BLE001)
112-114: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
112-114: Avoid specifying long messages outside the exception class
(TRY003)
133-133: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
181-181: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
222-225: Avoid specifying long messages outside the exception class
(TRY003)
wizard.py
331-331: String contains ambiguous ℹ (INFORMATION SOURCE). Did you mean i (LATIN SMALL LETTER I)?
(RUF001)
425-425: Possible hardcoded password assigned to: "neo4j_password"
(S105)
430-430: Possible hardcoded password assigned to: "neo4j_password"
(S105)
431-431: f-string without any placeholders
Remove extraneous f prefix
(F541)
465-465: f-string without any placeholders
Remove extraneous f prefix
(F541)
backends/advanced/src/advanced_omi_backend/services/memory/providers/chronicle.py
87-91: Abstract raise to an inner function
(TRY301)
87-91: Avoid specifying long messages outside the exception class
(TRY003)
93-96: Abstract raise to an inner function
(TRY301)
93-96: Avoid specifying long messages outside the exception class
(TRY003)
backends/advanced/src/advanced_omi_backend/speaker_recognition_client.py
172-172: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
177-177: Do not catch blind exception: Exception
(BLE001)
178-178: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
293-293: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
298-298: Do not catch blind exception: Exception
(BLE001)
299-299: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
backends/advanced/src/advanced_omi_backend/workers/speaker_jobs.py
235-235: Abstract raise to an inner function
(TRY301)
235-235: Avoid specifying long messages outside the exception class
(TRY003)
backends/advanced/init.py
698-698: f-string without any placeholders
Remove extraneous f prefix
(F541)
backends/advanced/src/advanced_omi_backend/services/memory/providers/llm_providers.py
193-197: Avoid specifying long messages outside the exception class
(TRY003)
200-203: Avoid specifying long messages outside the exception class
(TRY003)
317-317: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
317-317: f-string without any placeholders
Remove extraneous f prefix
(F541)
services.py
27-27: Do not catch blind exception: Exception
(BLE001)
113-113: String contains ambiguous ℹ (INFORMATION SOURCE). Did you mean i (LATIN SMALL LETTER I)?
(RUF001)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: robot-tests
🔇 Additional comments (71)
backends/advanced/src/advanced_omi_backend/workers/speaker_jobs.py (2)
61-76: Good error propagation for speaker service failures.This properly surfaces speaker service errors to the job orchestration layer instead of silently succeeding. The structured failure payload with
success: Falseand error details ensures downstream handling can react appropriately.
226-257: Well-structured error handling with appropriate severity differentiation.The logic correctly distinguishes between:
- Connection/timeout failures → raise exception to mark job as failed (cancels dependents)
- Processing errors → return structured failure (allows job completion with error info)
- Empty results → legitimate success with no speakers
This ensures RQ properly tracks job states and dependent jobs behave correctly.
tests/endpoints/system_admin_tests.robot (1)
82-82: Bug fix approved: Correct Python dictionary syntax.The change from
$response_data[speakers]to$response_data["speakers"]fixes a critical bug. The original would have raised aNameErrorbecausespeakers(without quotes) would be interpreted as an undefined Python variable within theisinstance()expression.tests/setup/test_data.py (1)
39-51: Data correctly transcribed from source file.The constant accurately matches the source data in
backends/advanced/tests/assets/expected_memories.json. The trailing period on the last memory string is consistent with the source and intentional.extras/speaker-recognition/run-test.sh (1)
20-21: LGTM! Good defensive implementation for test isolation.The changes correctly ensure Docker Compose project isolation:
- Line 21 uses inline
COMPOSE_PROJECT_NAMEso cleanup works even if triggered via signal trap before the main export- Line 128 exports the project name for all subsequent commands
- This prevents test containers from conflicting with development environment
The defensive approach (inline env var in cleanup + early export) is well thought out.
Also applies to: 127-129
tests/infrastructure/infra_tests.robot (2)
187-191: LGTM: Appropriate retry pattern for worker registration.The wait-and-retry pattern with clear logging is well-suited for infrastructure tests where worker registration timing can vary.
120-120: All tags used in this file (infra,queue,health,audio-streaming) are from the approved list intests/tags.mdand are correctly tab-separated. No action needed.backends/advanced/src/advanced_omi_backend/services/memory/providers/llm_providers.py (1)
11-11: LGTM! Imports and scaffolding updates are appropriate.The
asyncioimport supports the new timeout functionality, the spaCy TODO comments document the temporary disablement, and theModelDefimport aligns with the config-driven model registry refactor.Also applies to: 30-32, 37-37
backends/advanced/run-test.sh (5)
42-93: LGTM! Well-implemented environment variable precedence.The command-line override handling correctly implements priority: command-line > CI environment > .env.test > .env. The pattern of saving overrides before loading .env files, then restoring them, ensures command-line arguments take precedence as intended.
94-100: LGTM! Config file handling is clear and correct.The default CONFIG_FILE path correctly resolves to
config/config.ymlfrom the repository root, aligning with the centralized configuration structure introduced in this PR.
181-183: LGTM! Good practice for environment isolation.Using a unique Compose project name prevents conflicts between test and development containers, which is essential for reliable local testing.
207-211: Clarify TEST_MODE's purpose vs. CLEANUP_CONTAINERS.The comment states "Robot tests keep containers running (cleanup handled by run-test.sh)" when TEST_MODE=dev, but the actual cleanup behavior is controlled by the
CLEANUP_CONTAINERSflag (lines 222, 234). This could confuse users about which variable controls container lifecycle.Please verify:
- Does
TEST_MODE=devaffect Robot Framework test behavior (e.g., fixture teardown)?- Or is it only used as a marker in this script?
If TEST_MODE primarily affects Robot Framework's internal behavior, consider clarifying the comment to explain the distinction:
-# Configure Robot Framework test mode -# TEST_MODE=dev: Robot tests keep containers running (cleanup handled by run-test.sh) -# This allows CLEANUP_CONTAINERS flag to work as expected +# Configure Robot Framework test mode +# TEST_MODE=dev: Informs Robot Framework to skip internal cleanup +# Actual container cleanup is controlled by CLEANUP_CONTAINERS flag in run-test.sh export TEST_MODE=dev
212-231: LGTM! Robot Framework transition is properly implemented.The transition from pytest to Robot Framework is correctly executed:
- Tests run from the repository root where
backends/andtests/are siblings- 15-minute timeout accommodates comprehensive extraction
- Conditional cleanup on failure respects the
CLEANUP_CONTAINERSflag for debuggingtests/endpoints/rq_queue_tests.robot (2)
31-36: LGTM! Service restart logic is correct.The addition of
cwd=${BACKEND_DIR}ensures docker compose commands execute in the correct directory, and the timeout adjustments (30s stop, 60s start) are appropriate for service restart operations.
22-22: Verify the simplified compose file path is correct.The compose file path has been simplified from
backends/advanced/docker-compose-test.ymltodocker-compose-test.yml. Confirm this path resolves correctly relative to the${BACKEND_DIR}working directory specified in the docker compose commands, and ensure the file exists at the simplified location.backends/advanced/Docs/memory-configuration-guide.md (2)
9-12: LGTM! Documentation correctly reflects the new config path.The documentation has been properly updated to reference
config/config.ymlinstead ofconfig.yml, aligning with the centralized configuration directory structure introduced in this PR.
130-130: LGTM! Next Steps section updated correctly.The config path reference has been updated consistently with the rest of the documentation.
tests/Makefile (1)
34-34: LGTM! Test requirements properly integrated.The addition of
--with-requirements test-requirements.txtto all test targets ensures that test dependencies (including the newly addedwebsockets) are consistently loaded across all test suites.Also applies to: 42-42, 50-50, 58-58
tests/resources/queue_keywords.robot (1)
73-77: LGTM! Fast-fail behavior improves test feedback.The early exit when a job has already failed is a sensible optimization that prevents unnecessary polling and provides immediate feedback with the error details.
backends/advanced/README.md (1)
103-120: LGTM! Documentation clearly reflects the new config-driven testing approach.The updated examples demonstrate how to use different configuration files for testing, and the prerequisites are appropriately updated for the new workflow.
CLAUDE.md (2)
119-123: LGTM! Commands updated to reflect Robot Framework migration.The manual test execution and debugging commands correctly use the new Robot Framework syntax and paths.
393-393: Testing strategy documentation updated correctly.The reference now points to the Robot Framework-based integration tests.
tests/setup/test_env.py (1)
4-27: LGTM! Environment loading with clear precedence.The implementation correctly loads
.env.testbefore.envwithoverride=False, ensuring test-specific values take precedence while still allowing existing environment variables to take highest priority. The path constants are properly exported for Robot Framework use.tests/resources/memory_keywords.robot (2)
109-233: Well-implemented memory job waiting with comprehensive status handling.The keyword properly handles all job states (not_found, failed, completed, timeout) and returns a structured result dictionary. The polling approach with 5-second intervals and configurable timeout is appropriate for async job monitoring.
278-292: LGTM! Clean wrapper for memory quality verification.The keyword appropriately delegates to the similarity check and performs a clear assertion on the result.
tests/configs/parakeet-openai.yml (1)
1-73: Well-structured test configuration for hybrid Parakeet + OpenAI stack.The configuration clearly separates defaults, model definitions, and memory settings. The environment variable substitution syntax
${VAR:-default}is fully supported by the model registry loader (see_resolve_env()and_deep_resolve_env()inmodel_registry.py), which recursively resolves all placeholders before Pydantic validation. All environment variables used in this config are correctly formatted and will be resolved as intended.tests/run-robot-tests.sh (2)
38-57: LGTM! Robust path handling for Docker Compose.The absolute path conversion correctly handles relative CONFIG_FILE paths, ensuring Docker Compose resolves the config file consistently regardless of the compose file's location.
298-342: Excellent debugging instrumentation.The comprehensive log capture (container status, worker registration, service logs, and resource usage) will significantly improve troubleshooting capability. Error handling with
|| trueensures cleanup always completes even if individual captures fail.backends/advanced/src/advanced_omi_backend/services/memory/providers/chronicle.py (1)
87-96: LGTM! Enhanced error messages improve debuggability.The expanded error messages now include provider-specific context and actionable guidance for resolving connection failures. This aligns well with the PR's shift to config.yml-driven provider selection.
Note: The Ruff hints (TRY301, TRY003) are overly pedantic for initialization error handling where inline messages with context are more maintainable than custom exception classes.
backends/advanced/SETUP_SCRIPTS.md (1)
9-9: LGTM! Documentation correctly reflects new config path.The update accurately documents that memory configuration now resides in
config/config.yml, aligning with the PR's centralized configuration management approach.tests/setup/test_manager_keywords.robot (1)
65-66: LGTM! Path standardization improves maintainability.Replacing
${EXECDIR}/backends/advanced/data/...with${BACKEND_DIR}/data/...centralizes path management through the BACKEND_DIR variable, which aligns with the broader path management improvements introduced in test_env.py and setup_keywords.robot.Also applies to: 93-94
backends/advanced/.env.template (1)
48-56: LGTM! Clean separation of secrets and configuration.The restructured STT section properly separates API keys (secrets in .env) from provider selection (configuration in config.yml). The clear comments guide users to the appropriate location for each setting, which will reduce confusion during setup.
backends/advanced/src/advanced_omi_backend/routers/modules/health_routes.py (1)
131-135: LGTM! Registry-driven transcription provider display.The change correctly derives the transcription provider name from the registry with proper null guards for both
REGISTRYandget_default("stt")..github/workflows/README.md (1)
89-90: LGTM! Updated local testing instructions for Robot Framework.The documentation correctly reflects the migration from pytest to Robot Framework for integration tests.
tests/setup/teardown_keywords.robot (1)
41-41: LGTM! Centralized path variable usage.Using
${BACKEND_DIR}variable for the docker-compose path aligns with the centralized configuration approach.backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.py (1)
309-318: Provider validation logic is correct.The validation ensures only supported providers (
deepgramorparakeet) are used, with clear error messages for troubleshooting configuration issues.tests/resources/audio_keywords.robot (2)
98-99: Verify Robot Framework syntax fordict.get()in assertion message.The expression
${result.get('error_message', 'Unknown error')}may not evaluate correctly in Robot Framework's assertion message context. Consider using Collections library'sGet From Dictionarywith a default, or use a simpler message.🔎 Alternative approach
# Verify memory extraction succeeded - Should Be True ${result}[success] - ... Memory extraction failed: ${result.get('error_message', 'Unknown error')} + Should Be True ${result}[success] + ... Memory extraction failed - check job details for errorOr pre-fetch the error message:
${error_msg}= Get From Dictionary ${result} error_message default=Unknown error Should Be True ${result}[success] Memory extraction failed: ${error_msg}
72-113: Good keyword design for E2E memory validation.The keyword follows coding guidelines with a descriptive business-purpose name and inline assertions. It properly orchestrates the upload-to-memory-extraction flow and validates results comprehensively.
tests/integration/integration_test.robot (2)
14-15: LGTM! Resource imports for memory and queue keywords.Adding these resources enables the new memory validation test case.
132-166: Well-structured E2E test with memory validation.The test follows coding guidelines:
- Descriptive business-purpose name
- Inline verifications (lines 154-156, 160)
- Clear phase separation with logging
- Appropriate timeout for comprehensive pipeline testing
services.py (2)
18-29: LGTM! Config loading with graceful fallback.The function properly handles missing config files by returning
None, and catches exceptions to warn users without crashing. The broadExceptioncatch is appropriate here since we want graceful degradation for any YAML parsing issues.
91-113: Good config-driven Obsidian enablement with backward compatibility.The two-method approach (config.yml preferred, .env fallback) ensures backward compatibility while enabling the new centralized configuration. The logic correctly prioritizes config.yml over .env.
backends/advanced/docker-compose.yml (2)
63-63: Consistent with chronicle-backend service.The workers service volume mount mirrors the backend service change, maintaining consistency across containers.
15-15: Config path centralization approach is sound.The volume mount changes to
../../config/config.ymlcorrectly reference the shared configuration directory, and the removal of:roflags is intentional to enable UI-based configuration updates. The setup infrastructure is properly in place:wizard.pyhandles config file creation from theconfig/config.yml.template, andConfigManager(used bybackends/advanced/init.py) ensures the config directory and file exist with appropriate defaults before services start.backends/advanced/docker-compose-test.yml (4)
17-17: Good use of environment variable override for test config.The
${CONFIG_FILE:-...}pattern provides flexibility for CI to inject different test configurations while maintaining a sensible default.
56-57: Healthcheck adjustments improve CI reliability.Increasing retries from 5 to 10 and start_period from 30s to 60s helps accommodate slower CI runners and model initialization times. The dependency on
speaker-service-testwithservice_healthyensures proper startup ordering.Also applies to: 62-63
273-276: Bridge network isolation is appropriate for tests.Using a default bridge network instead of an external network ensures test isolation and avoids conflicts with development environments.
123-151: Speaker-service-test container is properly configured for CI testing.The Dockerfile correctly supports the
PYTORCH_CUDA_VERSION=cpubuild arg, using it to select appropriate extras in the uv dependency manager. The health check endpoint (port 8085) is properly exposed and the 60-second start period accommodates model loading. Volume mounts align with the service's directory structure for audio processing and model caching.Docs/getting-started.md (2)
182-184: Documentation updated for Robot Framework testing.The test command correctly uses
uv run robotwith appropriate flags. Based on learnings, this aligns with the project convention of usinguv runfor Python execution.
345-345: Config path references consistently updated.All references to the configuration file have been updated to
config/config.yml, maintaining consistency with the centralized configuration approach introduced in this PR.Also applies to: 544-548
tests/configs/parakeet-ollama.yml (2)
1-9: Well-structured test configuration for local/offline stack.The configuration properly defines defaults and model mappings for a fully local Parakeet + Ollama setup, enabling offline testing without API dependencies.
65-73: Memory extraction prompt is well-defined.The prompt clearly specifies the expected JSON output format with a "facts" array. The 1200-second timeout is generous but appropriate for test scenarios that may involve slower local models.
.github/workflows/robot-tests.yml (3)
31-31: HF_TOKEN validation added for speaker recognition models.This ensures the workflow fails fast if the HuggingFace token is missing, which is required for downloading speaker recognition models.
Also applies to: 42-48
80-86: Config setup simplified with direct file copy.Copying the test configuration directly from
tests/configs/deepgram-openai.ymlis cleaner than the previous approach and ensures consistency with local testing.
88-101: Test execution delegated to unified script.Using
run-robot-tests.shmirrors local development workflow and simplifies CI maintenance. TheCLEANUP_CONTAINERS=falsesetting is appropriate to preserve containers for log inspection on failure.backends/advanced/Docs/README.md (1)
16-16: Documentation path references consistently updated.All references to the configuration file have been updated to reflect the new
config/config.ymllocation. The file organization diagram accurately shows the new structure.Also applies to: 73-73, 133-133, 151-151
backends/advanced/Docs/quickstart.md (2)
180-182: Test documentation updated for Robot Framework.The command correctly uses
uv run robotwith appropriate output directory and log level flags, consistent with the project's testing guidelines.
343-343: Config path references consistently updated throughout.All memory configuration references now point to
config/config.yml, maintaining consistency with the centralized configuration approach.Also applies to: 542-545, 728-728
backends/advanced/start-workers.sh (2)
54-85: Config-driven STT provider selection looks good.The registry-based approach for determining which audio stream worker to start aligns well with the broader config-driven architecture. The Deepgram worker correctly requires both provider selection AND API key presence, while Parakeet only checks provider selection (acceptable since it's a local service without external credentials).
184-203: Improved graceful shutdown semantics.The shift from fail-fast
wait -nto monitoring-based waiting is a solid reliability improvement. The self-healing monitor handles worker restarts, and the explicit shutdown path ensures clean termination.backends/advanced/src/advanced_omi_backend/speaker_recognition_client.py (1)
28-67: Config-driven initialization is well-structured.The priority chain (param > config.yml > env var) for service_url and the clear enablement logic through the registry provides a clean configuration hierarchy. The early-return pattern for disabled states keeps the code readable.
wizard.py (3)
7-7: Good: Import moved to module level.The
shutilimport is now correctly placed at the top of the file, addressing the previous review comment.
317-331: Config file setup looks correct.The helper properly ensures the config directory exists before copying the template and handles both creation and skip scenarios with appropriate messaging.
400-434: Obsidian/Neo4j integration flow is well-structured.The interactive prompts follow the same pattern as other configuration sections, with proper EOFError handling and sensible defaults. The password validation (min 8 chars) is appropriate.
config/config.yml.template (2)
102-102: Good use of environment variable placeholder.The
${PARAKEET_ASR_URL:-172.17.0.1:8767}pattern allows runtime configuration while providing a sensible default.
208-219: New Obsidian and speaker_recognition sections are well-defined.The default values are sensible: Obsidian disabled by default (requires explicit opt-in), speaker_recognition enabled but with null service_url (effectively requiring configuration to work). The timeout values provide reasonable defaults.
tests/setup/setup_keywords.robot (2)
88-88: Good: Paths externalized to environment variables.Using
${BACKEND_DIR}instead of hardcoded paths improves flexibility and aligns with the environment-driven test configuration approach.Also applies to: 98-98, 123-123, 133-133
158-171: Speaker recognition keywords properly updated.The docker-compose commands now use
docker-compose-test.ymlwith the correctcwd=${SPEAKER_RECOGNITION_DIR}, consistent with other service management keywords.backends/advanced/init.py (3)
25-27: Path manipulation for ConfigManager import is acceptable.The
sys.path.insertis necessary here sinceconfig_manager.pyis at the repo root and this script runs frombackends/advanced. This is a reasonable approach for cross-directory imports.
43-51: ConfigManager integration looks correct.Initializing ConfigManager with the explicit service path ensures proper .env file handling, and the fallback to create a default structure handles missing config gracefully.
209-231: Transcription setup properly uses ConfigManager.The flow correctly updates config.yml via ConfigManager and reloads the data to stay in sync. The separation of API keys (in .env) from provider selection (in config.yml) is clean.
Comment on lines
+305
to
+320
| # Determine transcription provider from config.yml | ||
| from advanced_omi_backend.model_registry import get_models_registry | ||
|
|
||
| registry = get_models_registry() | ||
| if not registry: | ||
| raise ValueError("config.yml not found - cannot determine transcription provider") | ||
|
|
||
| stt_model = registry.get_default("stt") | ||
| if not stt_model: | ||
| raise ValueError("No default STT model configured in config.yml (defaults.stt)") | ||
|
|
||
| provider = stt_model.model_provider.lower() | ||
| if provider not in ["deepgram", "parakeet"]: | ||
| raise ValueError(f"Unsupported STT provider: {provider}. Expected: deepgram or parakeet") | ||
|
|
||
| application_logger.info(f"📋 Using STT provider: {provider} (model: {stt_model.name})") |
Contributor
There was a problem hiding this comment.
🛠️ Refactor suggestion | 🟠 Major
Move import to module level.
Per coding guidelines, all imports must be at the top of the file. The get_models_registry import on line 306 should be moved to the module-level imports section.
🔎 Proposed fix
Add to the imports at the top of the file (around line 22-26):
from advanced_omi_backend.model_registry import get_models_registryThen update lines 305-308:
# Determine transcription provider from config.yml
- from advanced_omi_backend.model_registry import get_models_registry
-
registry = get_models_registry()Committable suggestion skipped: line range outside the PR's diff.
🧰 Tools
🪛 Ruff (0.14.10)
310-310: Avoid specifying long messages outside the exception class
(TRY003)
314-314: Avoid specifying long messages outside the exception class
(TRY003)
318-318: Avoid specifying long messages outside the exception class
(TRY003)
🤖 Prompt for AI Agents
In
backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.py
around lines 305 to 320, the import for get_models_registry is done inside the
function; move that import to the module-level imports near the top of the file
(around lines 22–26) by adding "from advanced_omi_backend.model_registry import
get_models_registry" there and remove the local import at lines ~306; keep the
existing logic that calls get_models_registry and raises errors unchanged.
Comment on lines
+184
to
+187
| speaker_recognition: Dict[str, Any] = Field( | ||
| default_factory=dict, | ||
| description="Speaker recognition service configuration" | ||
| ) |
Contributor
There was a problem hiding this comment.
Add speaker_recognition to config loading.
The speaker_recognition field is defined in the AppModels schema but is not populated in the load_models_config function. This means it will always be an empty dict regardless of what's in config.yml.
🔎 Proposed fix to load speaker_recognition from config
In the load_models_config function, add loading of speaker_recognition settings similar to how memory settings are loaded:
# Extract sections
defaults = raw.get("defaults", {}) or {}
model_list = raw.get("models", []) or []
memory_settings = raw.get("memory", {}) or {}
+speaker_recognition_settings = raw.get("speaker_recognition", {}) or {}
# Parse and validate models using Pydantic
models: Dict[str, ModelDef] = {}Then update the AppModels instantiation:
# Create and cache registry
_REGISTRY = AppModels(
defaults=defaults,
models=models,
- memory=memory_settings
+ memory=memory_settings,
+ speaker_recognition=speaker_recognition_settings
)Committable suggestion skipped: line range outside the PR's diff.
🤖 Prompt for AI Agents
In backends/advanced/src/advanced_omi_backend/model_registry.py around lines
184-187, the AppModels schema defines speaker_recognition but load_models_config
never populates it so it remains an empty dict; update load_models_config to
read speaker_recognition from the models_config (e.g., speaker_recognition_cfg =
models_config.get("speaker_recognition", {})) and pass that value into the
AppModels constructor (speaker_recognition=speaker_recognition_cfg) just like
memory settings are loaded and passed through.
Comment on lines
+316
to
321
| except asyncio.TimeoutError: | ||
| memory_logger.error(f"OpenAI connection test timed out after 10s - check network connectivity and API endpoint") | ||
| return False | ||
| except Exception as e: | ||
| memory_logger.error(f"OpenAI connection test failed: {e}") | ||
| return False |
Contributor
There was a problem hiding this comment.
🛠️ Refactor suggestion | 🟠 Major
Use logging.exception to capture full stack traces.
Lines 317 and 320 use logging.error inside exception handlers, which loses stack trace information critical for debugging connection failures.
Based on learnings, prefer logging.exception() inside except blocks to automatically log the full stack trace. This improves debuggability when API connections fail.
🔎 Proposed fix
except asyncio.TimeoutError:
- memory_logger.error(f"OpenAI connection test timed out after 10s - check network connectivity and API endpoint")
+ memory_logger.exception("OpenAI connection test timed out after 10s - check network connectivity and API endpoint")
return False
except Exception as e:
- memory_logger.error(f"OpenAI connection test failed: {e}")
+ memory_logger.exception(f"OpenAI connection test failed: {e}")
return False📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| except asyncio.TimeoutError: | |
| memory_logger.error(f"OpenAI connection test timed out after 10s - check network connectivity and API endpoint") | |
| return False | |
| except Exception as e: | |
| memory_logger.error(f"OpenAI connection test failed: {e}") | |
| return False | |
| except asyncio.TimeoutError: | |
| memory_logger.exception("OpenAI connection test timed out after 10s - check network connectivity and API endpoint") | |
| return False | |
| except Exception as e: | |
| memory_logger.exception(f"OpenAI connection test failed: {e}") | |
| return False |
🧰 Tools
🪛 Ruff (0.14.10)
317-317: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
317-317: f-string without any placeholders
Remove extraneous f prefix
(F541)
319-319: Do not catch blind exception: Exception
(BLE001)
320-320: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
🤖 Prompt for AI Agents
In
backends/advanced/src/advanced_omi_backend/services/memory/providers/llm_providers.py
around lines 316 to 321, the exception handlers call memory_logger.error which
loses stack traces; change these to use memory_logger.exception (or
memory_logger.exception(..., exc_info=True)) with the same messages so the full
stack trace is recorded, and keep the existing return False behavior.
Comment on lines
219
to
222
| # Verify expected worker count (should be 7: 6 RQ workers + 1 audio persistence worker) | ||
| # Audio stream workers (Deepgram/Parakeet) are NOT RQ workers, so they don't register | ||
| Should Be True ${worker_count} >= 6 msg=Expected at least 6 RQ workers registered | ||
| Should Be True ${worker_count} <= 8 msg=Expected no more than 8 workers |
Contributor
There was a problem hiding this comment.
Clarify expected worker count in comment.
The comment states "should be 7: 6 RQ workers + 1 audio persistence worker" but the assertion allows a range of [6, 8]. This creates ambiguity about the actual expected behavior:
- Is 7 the typical count with tolerance for edge cases?
- Is the audio persistence worker optional or unreliable?
- Why is up to 8 workers acceptable?
Please update the comment to explain the range or adjust the assertion to match the stated expectation.
🤖 Prompt for AI Agents
In tests/infrastructure/infra_tests.robot around lines 219 to 222, the comment
says "should be 7: 6 RQ workers + 1 audio persistence worker" but the assertions
allow a range of 6–8, causing ambiguity; update the comment to explain that 7 is
the expected nominal count and the range 6–8 is tolerated to account for
optional/unregistered workers or transient registration timing, or tighten the
assertions to require exactly 7 if the audio persistence worker is
mandatory—choose one approach and make the comment and assertions consistent
(either document the tolerance and keep 6–8, or change the assertions to assert
equality to 7).
Comment on lines
+268
to
+270
| ${result_json}= Set Variable ${response.json()} | ||
| ${content}= Set Variable ${result_json}[choices][0][message][content] | ||
| ${similarity_result}= Evaluate json.loads("""${content}""") json |
Contributor
There was a problem hiding this comment.
Potential JSON parsing issue with special characters in OpenAI response.
Using json.loads("""${content}""") with Robot Framework's triple-quote syntax may fail if the OpenAI response content contains backslashes, nested quotes, or other characters that interfere with string interpolation.
🔎 Proposed fix using Evaluate with proper variable handling
${content}= Set Variable ${result_json}[choices][0][message][content]
- ${similarity_result}= Evaluate json.loads("""${content}""") json
+ ${similarity_result}= Evaluate json.loads($content) jsonUsing $content (without braces) passes the variable directly to the Python expression rather than string-interpolating it, avoiding escape sequence issues.
🤖 Prompt for AI Agents
In tests/resources/memory_keywords.robot around lines 268 to 270, the Evaluate
call uses triple-quoted string interpolation json.loads("""${content}""") which
can break when the OpenAI response contains backslashes or nested quotes; change
the call to pass the Robot variable directly into the Python expression (e.g.,
use json.loads($content) with the same json library argument) so the value is
handed to Python without Robot-side string interpolation.
Comment on lines
+55
to
+57
| Run Process rm -rf ${BACKEND_DIR}/data/test_mongo_data shell=True | ||
| Run Process rm -rf ${BACKEND_DIR}/data/test_qdrant_data shell=True | ||
| Run Process rm -rf ${BACKEND_DIR}/data/test_audio_chunks shell=True |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Search for all Run Process usages with shell=True in the test suite
rg -n "Run Process.*shell=True" tests/ --type robot -C 2Repository: chronicler-ai/chronicle
Length of output: 13707
🏁 Script executed:
# Also check if there are any other shell-related Process calls
rg -n "Run Process" tests/ --type robot | head -30Repository: chronicler-ai/chronicle
Length of output: 4966
🏁 Script executed:
# Look at the full context of the specific file mentioned
cat -n tests/setup/teardown_keywords.robot | sed -n '50,65p'Repository: chronicler-ai/chronicle
Length of output: 725
Fix Run Process usage: remove shell=True or use bash -c wrapper.
Lines 55-57 use separate arguments with shell=True, which doesn't work as expected with Robot Framework's Process library. When shell=True is specified, the entire command must be a single string or wrapped with bash -c.
Options:
-
Remove
shell=True(simplest—this command doesn't need it):Run Process rm -rf ${BACKEND_DIR}/data/test_mongo_data
-
Or use
bash -cwrapper (consistent with lines 65-66, 93-94 in the same suite):Run Process bash -c rm -rf ${BACKEND_DIR}/data/test_mongo_data shell=True
🤖 Prompt for AI Agents
In tests/setup/teardown_keywords.robot around lines 55 to 57, the Run Process
calls pass separate arguments while also using shell=True which is incorrect for
Robot Framework's Process library; update those three lines to either remove
shell=True and keep separate args (recommended), or wrap the command with a bash
-c invocation and keep shell=True to pass the entire command as a single
string—apply the same pattern used elsewhere in the suite for consistency.
🎉 Robot Framework Test ResultsStatus: ✅ All tests passed!
📊 View ReportsGitHub Pages (Live Reports): Download Artifacts:
|
- Added OBSIDIAN_ENABLED configuration to ChronicleSetup for improved feature toggling. - Introduced speaker_recognition configuration handling in model_registry.py to streamline model loading. - Refactored imports in deepgram.py to improve clarity and reduce redundancy.
Merged
🎉 Robot Framework Test ResultsStatus: ✅ All tests passed!
📊 View ReportsGitHub Pages (Live Reports): Download Artifacts:
|
AnkushMalaker
added a commit
that referenced
this pull request
Jan 3, 2026
* audio upload extension with gdrive credentials * FIX: API parameters * UPDATE: tmp files cleanup n code refactored as per review * REFACTOR: minor refactor as per review * REFACTOR: minor update as per review * UPDATE: gdrive sync logic * REFACTOR: code update as per gdrive and update credential client * REFACTOR: validation updated - as per review from CR * UPDATE: code has been refactore for UUID for diffrent audio upload sources * REFACTOR: updated code as per review * Update documentation and configuration to reflect the transition from 'friend-backend' to 'chronicle-backend' across various files, including setup instructions, Docker configurations, and service logs. * Update test script to use docker-compose-test.yml for all test-related operations * Added standard MIT license * Fix/cleanup model (#219) * refactor memory * add config * docstring * more cleanup * code quality * code quality * unused return * DOTTED GET * Refactor Docker and CI configurations - Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process. - Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services. - Added new dependencies for Google API clients in `uv.lock` to support upcoming features. * Update configuration files for model providers and Docker setup - Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram. - Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving. - Updated memory configuration endpoint to accept plain text for YAML input. * Update transcription job handling to format speaker IDs - Changed variable name from `speaker_name` to `speaker_id` for clarity. - Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling. * Remove loading of backend .env file in test environment setup - Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests. * Enhance configuration management and setup wizard - Updated README to reflect the new setup wizard process. - Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers. - Improved user feedback during configuration updates, including success messages for configuration file updates. - Enabled backup of existing `config.yml` before saving changes. * Enhance HTTPS configuration in setup wizard - Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates. - Improved user prompts for server IP/domain input during HTTPS setup. - Updated default behavior to use existing IP or localhost based on user input. - Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access. * Add source parameter to audio file writing in websocket controller - Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking. --------- Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com> * fix/broken-tests (#230) * refactor memory * add config * docstring * more cleanup * code quality * code quality * unused return * DOTTED GET * Refactor Docker and CI configurations - Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process. - Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services. - Added new dependencies for Google API clients in `uv.lock` to support upcoming features. * Update configuration files for model providers and Docker setup - Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram. - Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving. - Updated memory configuration endpoint to accept plain text for YAML input. * Update transcription job handling to format speaker IDs - Changed variable name from `speaker_name` to `speaker_id` for clarity. - Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling. * Remove loading of backend .env file in test environment setup - Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests. * Enhance configuration management and setup wizard - Updated README to reflect the new setup wizard process. - Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers. - Improved user feedback during configuration updates, including success messages for configuration file updates. - Enabled backup of existing `config.yml` before saving changes. * Enhance HTTPS configuration in setup wizard - Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates. - Improved user prompts for server IP/domain input during HTTPS setup. - Updated default behavior to use existing IP or localhost based on user input. - Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access. * Add source parameter to audio file writing in websocket controller - Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking. * Refactor error handling in system controller and update memory config routes - Replaced ValueError with HTTPException for better error handling in `save_diarization_settings` and `validate_memory_config` functions. - Introduced a new Pydantic model, `MemoryConfigRequest`, for validating memory configuration requests in the system routes. - Updated the `validate_memory_config` endpoint to accept the new request model, improving input handling and validation. --------- Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com> * Feat/add obsidian 3 (#233) * obsidian support * neo4j comment * cleanup code * unused line * unused line * Fix MemoryEntry object usage in chat service * comment * feat(obsidian): add obsidian memory search integration to chat * unit test * use rq * neo4j service * typefix * test fix * cleanup * cleanup * version changes * profile * remove unused imports * Refactor memory configuration validation endpoints - Removed the deprecated `validate_memory_config_raw` endpoint and replaced it with a new endpoint that accepts plain text for validation. - Updated the existing `validate_memory_config` endpoint to clarify that it now accepts JSON input. - Adjusted the API call in the frontend to point to the new validation endpoint. * Refactor health check model configuration loading - Updated the health check function to load model configuration from the models registry instead of the root config. - Improved error handling by logging warnings when model configuration loading fails. --------- Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com> * Update .gitignore to exclude all files in app/ios and app/android directories (#238) * fix: Copy full source code in speaker-recognition Dockerfile (#243) Adds COPY src/ src/ step after dependency installation to ensure all source files are available in the Docker image. This improves build caching while ensuring complete source code is present. * Enhance configuration management and add new setup scripts (#235) * Enhance configuration management and add new setup scripts - Updated .gitignore to include config.yml and its template. - Added config.yml.template for default configuration settings. - Introduced restart.sh script for service management. - Enhanced services.py to load config.yml and check for Obsidian/Neo4j integration. - Updated wizard.py to prompt for Obsidian/Neo4j configuration during setup and create config.yml from template if it doesn't exist. * Refactor transcription providers and enhance configuration management - Updated Docker Compose files to include the new Neo4j service configuration. - Added support for Obsidian/Neo4j integration in the setup process. - Refactored transcription providers to utilize a registry-driven approach for Deepgram and Parakeet. - Enhanced error handling and logging in transcription processes. - Improved environment variable management in test scripts to prioritize command-line overrides. - Removed deprecated Parakeet provider implementation and streamlined audio stream workers. * Update configuration management and enhance file structure, add test-matrix (#237) * Update configuration management and enhance file structure - Refactored configuration file paths to use a dedicated `config/` directory, including updates to `config.yml` and its template. - Modified service scripts to load the new configuration path for `config.yml`. - Enhanced `.gitignore` to include the new configuration files and templates. - Updated documentation to reflect changes in configuration file locations and usage. - Improved setup scripts to ensure proper creation and management of configuration files. - Added new test configurations for various provider combinations to streamline testing processes. * Add test requirements and clean up imports in wizard.py - Introduced a new `test-requirements.txt` file to manage testing dependencies. - Removed redundant import of `shutil` in `wizard.py` to improve code clarity. * Add ConfigManager for unified configuration management - Introduced a new `config_manager.py` module to handle reading and writing configurations from `config.yml` and `.env` files, ensuring backward compatibility. - Refactored `ChronicleSetup` in `backends/advanced/init.py` to utilize `ConfigManager` for loading and updating configurations, simplifying the setup process. - Removed redundant methods for loading and saving `config.yml` directly in `ChronicleSetup`, as these are now managed by `ConfigManager`. - Enhanced user feedback during configuration updates, including success messages for changes made to configuration files. * Refactor transcription provider configuration and enhance setup process - Updated `.env.template` to clarify speech-to-text configuration and removed deprecated options for Mistral. - Modified `docker-compose.yml` to streamline environment variable management by removing unused Mistral keys. - Enhanced `ChronicleSetup` in `init.py` to provide clearer user feedback and updated the transcription provider selection process to rely on `config.yml`. - Improved error handling in the websocket controller to determine the transcription provider from the model registry instead of environment variables. - Updated health check routes to reflect the new method of retrieving the transcription provider from `config.yml`. - Adjusted `config.yml.template` to include comments on transcription provider options for better user guidance. * Enhance ConfigManager with deep merge functionality - Updated the `update_memory_config` method to perform a deep merge of updates into the memory configuration, ensuring nested dictionaries are merged correctly. - Added a new `_deep_merge` method to handle recursive merging of dictionaries, improving configuration management capabilities. * Refactor run-test.sh and enhance memory extraction tests - Removed deprecated environment variable handling for TRANSCRIPTION_PROVIDER in `run-test.sh`, streamlining the configuration process. - Introduced a new `run-custom.sh` script for executing Robot tests with custom configurations, improving test flexibility. - Enhanced memory extraction tests in `audio_keywords.robot` and `memory_keywords.robot` to include detailed assertions and result handling. - Updated `queue_keywords.robot` to fail fast if a job is in a 'failed' state when expecting 'completed', improving error handling. - Refactored `test_env.py` to load environment variables with correct precedence, ensuring better configuration management. * unify tests to robot test, add some more clean up * Update health check configuration in docker-compose-test.yml (#241) - Increased the number of retries from 5 to 10 for improved resilience during service readiness checks. - Extended the start period from 30s to 60s to allow more time for services to initialize before health checks commence. * Add step to create test configuration file in robot-tests.yml - Introduced a new step in the GitHub Actions workflow to copy the test configuration file from tests/configs/deepgram-openai.yml to a new config/config.yml. - Added logging to confirm the creation of the test config file, improving visibility during the test setup process. * remove cache step since not required * coderabbit comments * Refactor ConfigManager error handling for configuration file loading - Updated the ConfigManager to raise RuntimeError exceptions when the configuration file is not found or is invalid, improving error visibility and user guidance. - Removed fallback behavior that previously returned the current directory, ensuring users are explicitly informed about missing or invalid configuration files. * Refactor _find_repo_root method in ConfigManager - Updated the _find_repo_root method to locate the repository root using the __file__ location instead of searching for config/config.yml, simplifying the logic and improving reliability. - Removed the previous error handling that raised a RuntimeError if the configuration file was not found, as the new approach assumes config_manager.py is always at the repo root. * Enhance speaker recognition service integration and error handling (#245) * Enhance speaker recognition service integration and error handling - Updated `docker-compose-test.yml` to enable speaker recognition in the test environment and added a new `speaker-service-test` service for testing purposes. - Refactored `run-test.sh` to improve the execution of Robot Framework tests from the repository root. - Enhanced error handling in `speaker_recognition_client.py` to return detailed error messages for connection issues. - Improved error logging in `speaker_jobs.py` to handle and report errors from the speaker recognition service more effectively. - Updated `Dockerfile` to copy the full source code after dependencies are cached, ensuring all necessary files are included in the image. * Remove integration tests workflow and enhance robot tests with HF_TOKEN verification - Deleted the `integration-tests.yml` workflow file to streamline CI processes. - Updated `robot-tests.yml` to include verification for the new `HF_TOKEN` secret, ensuring all required secrets are checked before running tests. * Fix key access in system admin tests to use string indexing for speakers data * Refactor Robot Framework tests and enhance error handling in memory services - Removed the creation of the test environment file from the GitHub Actions workflow to streamline setup. - Updated the Robot Framework tests to utilize a unified test script for improved consistency. - Enhanced error messages in the MemoryService class to provide more context on connection failures for LLM and vector store providers. - Added critical checks for API key presence in the OpenAIProvider class to ensure valid credentials are provided before proceeding. - Adjusted various test setup scripts to use a centralized BACKEND_DIR variable for better maintainability and clarity. * Refactor test container cleanup in run-robot-tests.sh - Updated the script to dynamically construct container names from docker-compose services, improving maintainability and reducing hardcoded values. - Enhanced the cleanup process for stuck test containers by utilizing the COMPOSE_PROJECT_NAME variable. * Enhance run-robot-tests.sh for improved logging and cleanup - Set absolute paths for consistent directory references to simplify navigation. - Capture container logs, status, and resource usage for better debugging. - Refactor cleanup process to utilize dynamic backend directory references, improving maintainability. - Ensure proper navigation back to the tests directory after operations. * Add speaker recognition configuration and update test script defaults - Introduced speaker recognition settings in config.yml.template, allowing for easy enable/disable and service URL configuration. - Updated run-robot-tests.sh to use a test-specific configuration file that disables speaker recognition for improved CI performance. - Modified deepgram-openai.yml to disable speaker recognition during CI tests to enhance execution speed. * Refactor speaker recognition configuration management - Updated docker-compose-test.yml to clarify speaker recognition settings, now controlled via config.yml for improved CI performance. - Enhanced model_registry.py to include a dedicated speaker_recognition field for better configuration handling. - Modified speaker_recognition_client.py to load configuration from config.yml, allowing for dynamic enabling/disabling of the speaker recognition service based on the configuration. * Add minimum worker count verification to infrastructure tests - Introduced a new keyword to verify that the minimum number of workers are registered, enhancing the robustness of health checks. - Updated the worker count validation test to include a wait mechanism for worker registration, improving test reliability. - Clarified comments regarding expected worker counts to reflect the distinction between RQ and audio stream workers. * Update configuration management and enhance model handling - Added OBSIDIAN_ENABLED configuration to ChronicleSetup for improved feature toggling. - Introduced speaker_recognition configuration handling in model_registry.py to streamline model loading. - Refactored imports in deepgram.py to improve clarity and reduce redundancy. * Refactor configuration management in wizard and ChronicleSetup (#246) * Refactor configuration management in wizard and ChronicleSetup - Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling. - Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth. - Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers. * Fix string formatting for error message in ChronicleSetup --------- Co-authored-by: 01PrathamS <pratham21btai35@karnavatiuniversity.edu.in> Co-authored-by: Stu Alexandere <thestumonkey@gmail.com> Co-authored-by: Stuart Alexander <stu@theawesome.co.uk> Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
This was referenced Jan 3, 2026
Closed
AnkushMalaker
added a commit
that referenced
this pull request
Jan 6, 2026
* audio upload extension with gdrive credentials * FIX: API parameters * UPDATE: tmp files cleanup n code refactored as per review * REFACTOR: minor refactor as per review * REFACTOR: minor update as per review * UPDATE: gdrive sync logic * REFACTOR: code update as per gdrive and update credential client * REFACTOR: validation updated - as per review from CR * UPDATE: code has been refactore for UUID for diffrent audio upload sources * REFACTOR: updated code as per review * Update documentation and configuration to reflect the transition from 'friend-backend' to 'chronicle-backend' across various files, including setup instructions, Docker configurations, and service logs. * Update test script to use docker-compose-test.yml for all test-related operations * Added standard MIT license * Fix/cleanup model (#219) * refactor memory * add config * docstring * more cleanup * code quality * code quality * unused return * DOTTED GET * Refactor Docker and CI configurations - Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process. - Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services. - Added new dependencies for Google API clients in `uv.lock` to support upcoming features. * Update configuration files for model providers and Docker setup - Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram. - Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving. - Updated memory configuration endpoint to accept plain text for YAML input. * Update transcription job handling to format speaker IDs - Changed variable name from `speaker_name` to `speaker_id` for clarity. - Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling. * Remove loading of backend .env file in test environment setup - Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests. * Enhance configuration management and setup wizard - Updated README to reflect the new setup wizard process. - Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers. - Improved user feedback during configuration updates, including success messages for configuration file updates. - Enabled backup of existing `config.yml` before saving changes. * Enhance HTTPS configuration in setup wizard - Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates. - Improved user prompts for server IP/domain input during HTTPS setup. - Updated default behavior to use existing IP or localhost based on user input. - Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access. * Add source parameter to audio file writing in websocket controller - Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking. --------- Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com> * fix/broken-tests (#230) * refactor memory * add config * docstring * more cleanup * code quality * code quality * unused return * DOTTED GET * Refactor Docker and CI configurations - Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process. - Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services. - Added new dependencies for Google API clients in `uv.lock` to support upcoming features. * Update configuration files for model providers and Docker setup - Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram. - Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving. - Updated memory configuration endpoint to accept plain text for YAML input. * Update transcription job handling to format speaker IDs - Changed variable name from `speaker_name` to `speaker_id` for clarity. - Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling. * Remove loading of backend .env file in test environment setup - Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests. * Enhance configuration management and setup wizard - Updated README to reflect the new setup wizard process. - Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers. - Improved user feedback during configuration updates, including success messages for configuration file updates. - Enabled backup of existing `config.yml` before saving changes. * Enhance HTTPS configuration in setup wizard - Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates. - Improved user prompts for server IP/domain input during HTTPS setup. - Updated default behavior to use existing IP or localhost based on user input. - Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access. * Add source parameter to audio file writing in websocket controller - Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking. * Refactor error handling in system controller and update memory config routes - Replaced ValueError with HTTPException for better error handling in `save_diarization_settings` and `validate_memory_config` functions. - Introduced a new Pydantic model, `MemoryConfigRequest`, for validating memory configuration requests in the system routes. - Updated the `validate_memory_config` endpoint to accept the new request model, improving input handling and validation. --------- Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com> * Feat/add obsidian 3 (#233) * obsidian support * neo4j comment * cleanup code * unused line * unused line * Fix MemoryEntry object usage in chat service * comment * feat(obsidian): add obsidian memory search integration to chat * unit test * use rq * neo4j service * typefix * test fix * cleanup * cleanup * version changes * profile * remove unused imports * Refactor memory configuration validation endpoints - Removed the deprecated `validate_memory_config_raw` endpoint and replaced it with a new endpoint that accepts plain text for validation. - Updated the existing `validate_memory_config` endpoint to clarify that it now accepts JSON input. - Adjusted the API call in the frontend to point to the new validation endpoint. * Refactor health check model configuration loading - Updated the health check function to load model configuration from the models registry instead of the root config. - Improved error handling by logging warnings when model configuration loading fails. --------- Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com> * Update .gitignore to exclude all files in app/ios and app/android directories (#238) * fix: Copy full source code in speaker-recognition Dockerfile (#243) Adds COPY src/ src/ step after dependency installation to ensure all source files are available in the Docker image. This improves build caching while ensuring complete source code is present. * Enhance configuration management and add new setup scripts (#235) * Enhance configuration management and add new setup scripts - Updated .gitignore to include config.yml and its template. - Added config.yml.template for default configuration settings. - Introduced restart.sh script for service management. - Enhanced services.py to load config.yml and check for Obsidian/Neo4j integration. - Updated wizard.py to prompt for Obsidian/Neo4j configuration during setup and create config.yml from template if it doesn't exist. * Refactor transcription providers and enhance configuration management - Updated Docker Compose files to include the new Neo4j service configuration. - Added support for Obsidian/Neo4j integration in the setup process. - Refactored transcription providers to utilize a registry-driven approach for Deepgram and Parakeet. - Enhanced error handling and logging in transcription processes. - Improved environment variable management in test scripts to prioritize command-line overrides. - Removed deprecated Parakeet provider implementation and streamlined audio stream workers. * Update configuration management and enhance file structure, add test-matrix (#237) * Update configuration management and enhance file structure - Refactored configuration file paths to use a dedicated `config/` directory, including updates to `config.yml` and its template. - Modified service scripts to load the new configuration path for `config.yml`. - Enhanced `.gitignore` to include the new configuration files and templates. - Updated documentation to reflect changes in configuration file locations and usage. - Improved setup scripts to ensure proper creation and management of configuration files. - Added new test configurations for various provider combinations to streamline testing processes. * Add test requirements and clean up imports in wizard.py - Introduced a new `test-requirements.txt` file to manage testing dependencies. - Removed redundant import of `shutil` in `wizard.py` to improve code clarity. * Add ConfigManager for unified configuration management - Introduced a new `config_manager.py` module to handle reading and writing configurations from `config.yml` and `.env` files, ensuring backward compatibility. - Refactored `ChronicleSetup` in `backends/advanced/init.py` to utilize `ConfigManager` for loading and updating configurations, simplifying the setup process. - Removed redundant methods for loading and saving `config.yml` directly in `ChronicleSetup`, as these are now managed by `ConfigManager`. - Enhanced user feedback during configuration updates, including success messages for changes made to configuration files. * Refactor transcription provider configuration and enhance setup process - Updated `.env.template` to clarify speech-to-text configuration and removed deprecated options for Mistral. - Modified `docker-compose.yml` to streamline environment variable management by removing unused Mistral keys. - Enhanced `ChronicleSetup` in `init.py` to provide clearer user feedback and updated the transcription provider selection process to rely on `config.yml`. - Improved error handling in the websocket controller to determine the transcription provider from the model registry instead of environment variables. - Updated health check routes to reflect the new method of retrieving the transcription provider from `config.yml`. - Adjusted `config.yml.template` to include comments on transcription provider options for better user guidance. * Enhance ConfigManager with deep merge functionality - Updated the `update_memory_config` method to perform a deep merge of updates into the memory configuration, ensuring nested dictionaries are merged correctly. - Added a new `_deep_merge` method to handle recursive merging of dictionaries, improving configuration management capabilities. * Refactor run-test.sh and enhance memory extraction tests - Removed deprecated environment variable handling for TRANSCRIPTION_PROVIDER in `run-test.sh`, streamlining the configuration process. - Introduced a new `run-custom.sh` script for executing Robot tests with custom configurations, improving test flexibility. - Enhanced memory extraction tests in `audio_keywords.robot` and `memory_keywords.robot` to include detailed assertions and result handling. - Updated `queue_keywords.robot` to fail fast if a job is in a 'failed' state when expecting 'completed', improving error handling. - Refactored `test_env.py` to load environment variables with correct precedence, ensuring better configuration management. * unify tests to robot test, add some more clean up * Update health check configuration in docker-compose-test.yml (#241) - Increased the number of retries from 5 to 10 for improved resilience during service readiness checks. - Extended the start period from 30s to 60s to allow more time for services to initialize before health checks commence. * Add step to create test configuration file in robot-tests.yml - Introduced a new step in the GitHub Actions workflow to copy the test configuration file from tests/configs/deepgram-openai.yml to a new config/config.yml. - Added logging to confirm the creation of the test config file, improving visibility during the test setup process. * remove cache step since not required * coderabbit comments * Refactor ConfigManager error handling for configuration file loading - Updated the ConfigManager to raise RuntimeError exceptions when the configuration file is not found or is invalid, improving error visibility and user guidance. - Removed fallback behavior that previously returned the current directory, ensuring users are explicitly informed about missing or invalid configuration files. * Refactor _find_repo_root method in ConfigManager - Updated the _find_repo_root method to locate the repository root using the __file__ location instead of searching for config/config.yml, simplifying the logic and improving reliability. - Removed the previous error handling that raised a RuntimeError if the configuration file was not found, as the new approach assumes config_manager.py is always at the repo root. * Enhance speaker recognition service integration and error handling (#245) * Enhance speaker recognition service integration and error handling - Updated `docker-compose-test.yml` to enable speaker recognition in the test environment and added a new `speaker-service-test` service for testing purposes. - Refactored `run-test.sh` to improve the execution of Robot Framework tests from the repository root. - Enhanced error handling in `speaker_recognition_client.py` to return detailed error messages for connection issues. - Improved error logging in `speaker_jobs.py` to handle and report errors from the speaker recognition service more effectively. - Updated `Dockerfile` to copy the full source code after dependencies are cached, ensuring all necessary files are included in the image. * Remove integration tests workflow and enhance robot tests with HF_TOKEN verification - Deleted the `integration-tests.yml` workflow file to streamline CI processes. - Updated `robot-tests.yml` to include verification for the new `HF_TOKEN` secret, ensuring all required secrets are checked before running tests. * Fix key access in system admin tests to use string indexing for speakers data * Refactor Robot Framework tests and enhance error handling in memory services - Removed the creation of the test environment file from the GitHub Actions workflow to streamline setup. - Updated the Robot Framework tests to utilize a unified test script for improved consistency. - Enhanced error messages in the MemoryService class to provide more context on connection failures for LLM and vector store providers. - Added critical checks for API key presence in the OpenAIProvider class to ensure valid credentials are provided before proceeding. - Adjusted various test setup scripts to use a centralized BACKEND_DIR variable for better maintainability and clarity. * Refactor test container cleanup in run-robot-tests.sh - Updated the script to dynamically construct container names from docker-compose services, improving maintainability and reducing hardcoded values. - Enhanced the cleanup process for stuck test containers by utilizing the COMPOSE_PROJECT_NAME variable. * Enhance run-robot-tests.sh for improved logging and cleanup - Set absolute paths for consistent directory references to simplify navigation. - Capture container logs, status, and resource usage for better debugging. - Refactor cleanup process to utilize dynamic backend directory references, improving maintainability. - Ensure proper navigation back to the tests directory after operations. * Add speaker recognition configuration and update test script defaults - Introduced speaker recognition settings in config.yml.template, allowing for easy enable/disable and service URL configuration. - Updated run-robot-tests.sh to use a test-specific configuration file that disables speaker recognition for improved CI performance. - Modified deepgram-openai.yml to disable speaker recognition during CI tests to enhance execution speed. * Refactor speaker recognition configuration management - Updated docker-compose-test.yml to clarify speaker recognition settings, now controlled via config.yml for improved CI performance. - Enhanced model_registry.py to include a dedicated speaker_recognition field for better configuration handling. - Modified speaker_recognition_client.py to load configuration from config.yml, allowing for dynamic enabling/disabling of the speaker recognition service based on the configuration. * Add minimum worker count verification to infrastructure tests - Introduced a new keyword to verify that the minimum number of workers are registered, enhancing the robustness of health checks. - Updated the worker count validation test to include a wait mechanism for worker registration, improving test reliability. - Clarified comments regarding expected worker counts to reflect the distinction between RQ and audio stream workers. * Update configuration management and enhance model handling - Added OBSIDIAN_ENABLED configuration to ChronicleSetup for improved feature toggling. - Introduced speaker_recognition configuration handling in model_registry.py to streamline model loading. - Refactored imports in deepgram.py to improve clarity and reduce redundancy. * Refactor configuration management in wizard and ChronicleSetup (#246) * Refactor configuration management in wizard and ChronicleSetup - Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling. - Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth. - Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers. * Fix string formatting for error message in ChronicleSetup * added JWT issuers for audience auth for service interop and shared us… (#250) * added JWT issuers for audience auth for service interop and shared user accounts * amended default value in line wioth code --------- Co-authored-by: 01PrathamS <pratham21btai35@karnavatiuniversity.edu.in> Co-authored-by: Stu Alexandere <thestumonkey@gmail.com> Co-authored-by: Stuart Alexander <stu@theawesome.co.uk> Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
AnkushMalaker
added a commit
that referenced
this pull request
Feb 19, 2026
* audio upload extension with gdrive credentials
* FIX: API parameters
* UPDATE: tmp files cleanup n code refactored as per review
* REFACTOR: minor refactor as per review
* REFACTOR: minor update as per review
* UPDATE: gdrive sync logic
* REFACTOR: code update as per gdrive and update credential client
* REFACTOR: validation updated - as per review from CR
* UPDATE: code has been refactore for UUID for diffrent audio upload sources
* REFACTOR: updated code as per review
* Update documentation and configuration to reflect the transition from 'friend-backend' to 'chronicle-backend' across various files, including setup instructions, Docker configurations, and service logs.
* Update test script to use docker-compose-test.yml for all test-related operations
* Added standard MIT license
* Fix/cleanup model (#219)
* refactor memory
* add config
* docstring
* more cleanup
* code quality
* code quality
* unused return
* DOTTED GET
* Refactor Docker and CI configurations
- Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process.
- Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services.
- Added new dependencies for Google API clients in `uv.lock` to support upcoming features.
* Update configuration files for model providers and Docker setup
- Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram.
- Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving.
- Updated memory configuration endpoint to accept plain text for YAML input.
* Update transcription job handling to format speaker IDs
- Changed variable name from `speaker_name` to `speaker_id` for clarity.
- Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling.
* Remove loading of backend .env file in test environment setup
- Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests.
* Enhance configuration management and setup wizard
- Updated README to reflect the new setup wizard process.
- Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers.
- Improved user feedback during configuration updates, including success messages for configuration file updates.
- Enabled backup of existing `config.yml` before saving changes.
* Enhance HTTPS configuration in setup wizard
- Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates.
- Improved user prompts for server IP/domain input during HTTPS setup.
- Updated default behavior to use existing IP or localhost based on user input.
- Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access.
* Add source parameter to audio file writing in websocket controller
- Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking.
---------
Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
* fix/broken-tests (#230)
* refactor memory
* add config
* docstring
* more cleanup
* code quality
* code quality
* unused return
* DOTTED GET
* Refactor Docker and CI configurations
- Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process.
- Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services.
- Added new dependencies for Google API clients in `uv.lock` to support upcoming features.
* Update configuration files for model providers and Docker setup
- Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram.
- Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving.
- Updated memory configuration endpoint to accept plain text for YAML input.
* Update transcription job handling to format speaker IDs
- Changed variable name from `speaker_name` to `speaker_id` for clarity.
- Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling.
* Remove loading of backend .env file in test environment setup
- Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests.
* Enhance configuration management and setup wizard
- Updated README to reflect the new setup wizard process.
- Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers.
- Improved user feedback during configuration updates, including success messages for configuration file updates.
- Enabled backup of existing `config.yml` before saving changes.
* Enhance HTTPS configuration in setup wizard
- Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates.
- Improved user prompts for server IP/domain input during HTTPS setup.
- Updated default behavior to use existing IP or localhost based on user input.
- Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access.
* Add source parameter to audio file writing in websocket controller
- Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking.
* Refactor error handling in system controller and update memory config routes
- Replaced ValueError with HTTPException for better error handling in `save_diarization_settings` and `validate_memory_config` functions.
- Introduced a new Pydantic model, `MemoryConfigRequest`, for validating memory configuration requests in the system routes.
- Updated the `validate_memory_config` endpoint to accept the new request model, improving input handling and validation.
---------
Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
* Feat/add obsidian 3 (#233)
* obsidian support
* neo4j comment
* cleanup code
* unused line
* unused line
* Fix MemoryEntry object usage in chat service
* comment
* feat(obsidian): add obsidian memory search integration to chat
* unit test
* use rq
* neo4j service
* typefix
* test fix
* cleanup
* cleanup
* version changes
* profile
* remove unused imports
* Refactor memory configuration validation endpoints
- Removed the deprecated `validate_memory_config_raw` endpoint and replaced it with a new endpoint that accepts plain text for validation.
- Updated the existing `validate_memory_config` endpoint to clarify that it now accepts JSON input.
- Adjusted the API call in the frontend to point to the new validation endpoint.
* Refactor health check model configuration loading
- Updated the health check function to load model configuration from the models registry instead of the root config.
- Improved error handling by logging warnings when model configuration loading fails.
---------
Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
* Update .gitignore to exclude all files in app/ios and app/android directories (#238)
* fix: Copy full source code in speaker-recognition Dockerfile (#243)
Adds COPY src/ src/ step after dependency installation to ensure
all source files are available in the Docker image. This improves
build caching while ensuring complete source code is present.
* Enhance configuration management and add new setup scripts (#235)
* Enhance configuration management and add new setup scripts
- Updated .gitignore to include config.yml and its template.
- Added config.yml.template for default configuration settings.
- Introduced restart.sh script for service management.
- Enhanced services.py to load config.yml and check for Obsidian/Neo4j integration.
- Updated wizard.py to prompt for Obsidian/Neo4j configuration during setup and create config.yml from template if it doesn't exist.
* Refactor transcription providers and enhance configuration management
- Updated Docker Compose files to include the new Neo4j service configuration.
- Added support for Obsidian/Neo4j integration in the setup process.
- Refactored transcription providers to utilize a registry-driven approach for Deepgram and Parakeet.
- Enhanced error handling and logging in transcription processes.
- Improved environment variable management in test scripts to prioritize command-line overrides.
- Removed deprecated Parakeet provider implementation and streamlined audio stream workers.
* Update configuration management and enhance file structure, add test-matrix (#237)
* Update configuration management and enhance file structure
- Refactored configuration file paths to use a dedicated `config/` directory, including updates to `config.yml` and its template.
- Modified service scripts to load the new configuration path for `config.yml`.
- Enhanced `.gitignore` to include the new configuration files and templates.
- Updated documentation to reflect changes in configuration file locations and usage.
- Improved setup scripts to ensure proper creation and management of configuration files.
- Added new test configurations for various provider combinations to streamline testing processes.
* Add test requirements and clean up imports in wizard.py
- Introduced a new `test-requirements.txt` file to manage testing dependencies.
- Removed redundant import of `shutil` in `wizard.py` to improve code clarity.
* Add ConfigManager for unified configuration management
- Introduced a new `config_manager.py` module to handle reading and writing configurations from `config.yml` and `.env` files, ensuring backward compatibility.
- Refactored `ChronicleSetup` in `backends/advanced/init.py` to utilize `ConfigManager` for loading and updating configurations, simplifying the setup process.
- Removed redundant methods for loading and saving `config.yml` directly in `ChronicleSetup`, as these are now managed by `ConfigManager`.
- Enhanced user feedback during configuration updates, including success messages for changes made to configuration files.
* Refactor transcription provider configuration and enhance setup process
- Updated `.env.template` to clarify speech-to-text configuration and removed deprecated options for Mistral.
- Modified `docker-compose.yml` to streamline environment variable management by removing unused Mistral keys.
- Enhanced `ChronicleSetup` in `init.py` to provide clearer user feedback and updated the transcription provider selection process to rely on `config.yml`.
- Improved error handling in the websocket controller to determine the transcription provider from the model registry instead of environment variables.
- Updated health check routes to reflect the new method of retrieving the transcription provider from `config.yml`.
- Adjusted `config.yml.template` to include comments on transcription provider options for better user guidance.
* Enhance ConfigManager with deep merge functionality
- Updated the `update_memory_config` method to perform a deep merge of updates into the memory configuration, ensuring nested dictionaries are merged correctly.
- Added a new `_deep_merge` method to handle recursive merging of dictionaries, improving configuration management capabilities.
* Refactor run-test.sh and enhance memory extraction tests
- Removed deprecated environment variable handling for TRANSCRIPTION_PROVIDER in `run-test.sh`, streamlining the configuration process.
- Introduced a new `run-custom.sh` script for executing Robot tests with custom configurations, improving test flexibility.
- Enhanced memory extraction tests in `audio_keywords.robot` and `memory_keywords.robot` to include detailed assertions and result handling.
- Updated `queue_keywords.robot` to fail fast if a job is in a 'failed' state when expecting 'completed', improving error handling.
- Refactored `test_env.py` to load environment variables with correct precedence, ensuring better configuration management.
* unify tests to robot test, add some more clean up
* Update health check configuration in docker-compose-test.yml (#241)
- Increased the number of retries from 5 to 10 for improved resilience during service readiness checks.
- Extended the start period from 30s to 60s to allow more time for services to initialize before health checks commence.
* Add step to create test configuration file in robot-tests.yml
- Introduced a new step in the GitHub Actions workflow to copy the test configuration file from tests/configs/deepgram-openai.yml to a new config/config.yml.
- Added logging to confirm the creation of the test config file, improving visibility during the test setup process.
* remove cache step since not required
* coderabbit comments
* Refactor ConfigManager error handling for configuration file loading
- Updated the ConfigManager to raise RuntimeError exceptions when the configuration file is not found or is invalid, improving error visibility and user guidance.
- Removed fallback behavior that previously returned the current directory, ensuring users are explicitly informed about missing or invalid configuration files.
* Refactor _find_repo_root method in ConfigManager
- Updated the _find_repo_root method to locate the repository root using the __file__ location instead of searching for config/config.yml, simplifying the logic and improving reliability.
- Removed the previous error handling that raised a RuntimeError if the configuration file was not found, as the new approach assumes config_manager.py is always at the repo root.
* Enhance speaker recognition service integration and error handling (#245)
* Enhance speaker recognition service integration and error handling
- Updated `docker-compose-test.yml` to enable speaker recognition in the test environment and added a new `speaker-service-test` service for testing purposes.
- Refactored `run-test.sh` to improve the execution of Robot Framework tests from the repository root.
- Enhanced error handling in `speaker_recognition_client.py` to return detailed error messages for connection issues.
- Improved error logging in `speaker_jobs.py` to handle and report errors from the speaker recognition service more effectively.
- Updated `Dockerfile` to copy the full source code after dependencies are cached, ensuring all necessary files are included in the image.
* Remove integration tests workflow and enhance robot tests with HF_TOKEN verification
- Deleted the `integration-tests.yml` workflow file to streamline CI processes.
- Updated `robot-tests.yml` to include verification for the new `HF_TOKEN` secret, ensuring all required secrets are checked before running tests.
* Fix key access in system admin tests to use string indexing for speakers data
* Refactor Robot Framework tests and enhance error handling in memory services
- Removed the creation of the test environment file from the GitHub Actions workflow to streamline setup.
- Updated the Robot Framework tests to utilize a unified test script for improved consistency.
- Enhanced error messages in the MemoryService class to provide more context on connection failures for LLM and vector store providers.
- Added critical checks for API key presence in the OpenAIProvider class to ensure valid credentials are provided before proceeding.
- Adjusted various test setup scripts to use a centralized BACKEND_DIR variable for better maintainability and clarity.
* Refactor test container cleanup in run-robot-tests.sh
- Updated the script to dynamically construct container names from docker-compose services, improving maintainability and reducing hardcoded values.
- Enhanced the cleanup process for stuck test containers by utilizing the COMPOSE_PROJECT_NAME variable.
* Enhance run-robot-tests.sh for improved logging and cleanup
- Set absolute paths for consistent directory references to simplify navigation.
- Capture container logs, status, and resource usage for better debugging.
- Refactor cleanup process to utilize dynamic backend directory references, improving maintainability.
- Ensure proper navigation back to the tests directory after operations.
* Add speaker recognition configuration and update test script defaults
- Introduced speaker recognition settings in config.yml.template, allowing for easy enable/disable and service URL configuration.
- Updated run-robot-tests.sh to use a test-specific configuration file that disables speaker recognition for improved CI performance.
- Modified deepgram-openai.yml to disable speaker recognition during CI tests to enhance execution speed.
* Refactor speaker recognition configuration management
- Updated docker-compose-test.yml to clarify speaker recognition settings, now controlled via config.yml for improved CI performance.
- Enhanced model_registry.py to include a dedicated speaker_recognition field for better configuration handling.
- Modified speaker_recognition_client.py to load configuration from config.yml, allowing for dynamic enabling/disabling of the speaker recognition service based on the configuration.
* Add minimum worker count verification to infrastructure tests
- Introduced a new keyword to verify that the minimum number of workers are registered, enhancing the robustness of health checks.
- Updated the worker count validation test to include a wait mechanism for worker registration, improving test reliability.
- Clarified comments regarding expected worker counts to reflect the distinction between RQ and audio stream workers.
* Update configuration management and enhance model handling
- Added OBSIDIAN_ENABLED configuration to ChronicleSetup for improved feature toggling.
- Introduced speaker_recognition configuration handling in model_registry.py to streamline model loading.
- Refactored imports in deepgram.py to improve clarity and reduce redundancy.
* Refactor configuration management in wizard and ChronicleSetup (#246)
* Refactor configuration management in wizard and ChronicleSetup
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling.
- Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth.
- Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
* Fix string formatting for error message in ChronicleSetup
* added JWT issuers for audience auth for service interop and shared us… (#250)
* added JWT issuers for audience auth for service interop and shared user accounts
* amended default value in line wioth code
* Feat/edit chat system prompt (#247)
* Refactor configuration management in wizard and ChronicleSetup
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling.
- Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth.
- Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
* Fix string formatting for error message in ChronicleSetup
* Enhance chat configuration management and UI integration
- Updated `services.py` to allow service restart with an option to recreate containers, addressing WSL2 bind mount issues.
- Added new chat configuration management functions in `system_controller.py` for loading, saving, and validating chat prompts.
- Introduced `ChatSettings` component in the web UI for admin users to manage chat configurations easily.
- Updated API service methods in `api.ts` to support chat configuration endpoints.
- Integrated chat settings into the system management page for better accessibility.
* Refactor backend shutdown process and enhance chat service configuration logging
- Updated `start.sh` to improve shutdown handling by explicitly killing the backend process if running.
- Modified `chat_service.py` to enhance logging for loading chat system prompts, providing clearer feedback on configuration usage.
- Added a new `chat` field in `model_registry.py` for better chat service configuration management.
- Updated vector store query parameters in `vector_stores.py` for improved clarity and functionality.
- Enhanced the chat component in the web UI to conditionally auto-scroll based on message sending status.
* Return JSONResponse instead of raw result
* Refactor headers creation in system admin tests
* Make config.yml writable for admin updates
* Docs consolidation (#257)
* Enhance setup documentation and convenience scripts
- Updated the interactive setup wizard instructions to recommend using the convenience script `./wizard.sh` for easier configuration.
- Added detailed instructions for uploading and processing existing audio files via the API, including example commands for single and multiple file uploads.
- Introduced a new section on HAVPE relay configuration for ESP32 audio streaming, providing environment variable setup and command examples.
- Clarified the distributed deployment setup, including GPU and backend separation instructions, and added benefits of using Tailscale for networking.
- Removed outdated `getting-started.md` and `SETUP_SCRIPTS.md` files to streamline documentation and avoid redundancy.
* Update setup instructions and enhance service management scripts
- Replaced direct command instructions with convenience scripts (`./wizard.sh` and `./start.sh`) for easier setup and service management.
- Added detailed usage of convenience scripts for checking service status, restarting, and stopping services.
- Clarified the distinction between convenience scripts and direct command usage for improved user guidance.
* Update speaker recognition models and documentation
- Changed the speaker diarization model from `pyannote/speaker-diarization-3.1` to `pyannote/speaker-diarization-community-1` across multiple files for consistency.
- Updated README files to reflect the new model and its usage instructions, ensuring users have the correct links and information for setup.
- Enhanced clarity in configuration settings related to speaker recognition.
* Docs consolidation (#258)
* Enhance setup documentation and convenience scripts
- Updated the interactive setup wizard instructions to recommend using the convenience script `./wizard.sh` for easier configuration.
- Added detailed instructions for uploading and processing existing audio files via the API, including example commands for single and multiple file uploads.
- Introduced a new section on HAVPE relay configuration for ESP32 audio streaming, providing environment variable setup and command examples.
- Clarified the distributed deployment setup, including GPU and backend separation instructions, and added benefits of using Tailscale for networking.
- Removed outdated `getting-started.md` and `SETUP_SCRIPTS.md` files to streamline documentation and avoid redundancy.
* Update setup instructions and enhance service management scripts
- Replaced direct command instructions with convenience scripts (`./wizard.sh` and `./start.sh`) for easier setup and service management.
- Added detailed usage of convenience scripts for checking service status, restarting, and stopping services.
- Clarified the distinction between convenience scripts and direct command usage for improved user guidance.
* Update speaker recognition models and documentation
- Changed the speaker diarization model from `pyannote/speaker-diarization-3.1` to `pyannote/speaker-diarization-community-1` across multiple files for consistency.
- Updated README files to reflect the new model and its usage instructions, ensuring users have the correct links and information for setup.
- Enhanced clarity in configuration settings related to speaker recognition.
* Enhance transcription provider selection and update HTTPS documentation
- Added a new function in `wizard.py` to prompt users for their preferred transcription provider, allowing options for Deepgram, Parakeet ASR, or none.
- Updated the service setup logic to automatically include ASR services if Parakeet is selected.
- Introduced a new documentation file on SSL certificates and HTTPS setup, detailing the importance of HTTPS for secure connections and microphone access.
- Removed outdated HTTPS setup documentation from `backends/advanced/Docs/HTTPS_SETUP.md` to streamline resources.
* Remove HTTPS setup scripts and related configurations
- Deleted `init-https.sh`, `setup-https.sh`, and `nginx.conf.template` as part of the transition to a new HTTPS setup process.
- Updated `README.md` to reflect the new automatic HTTPS configuration via the setup wizard.
- Adjusted `init.py` to remove references to the deleted HTTPS scripts and ensure proper handling of Caddyfile generation for SSL.
- Streamlined documentation to clarify the new approach for HTTPS setup and configuration management.
* Update quickstart.md (#268)
* v0.2 (#279)
* Refactor configuration management in wizard and ChronicleSetup
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling.
- Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth.
- Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
* Fix string formatting for error message in ChronicleSetup
* Enhance chat configuration management and UI integration
- Updated `services.py` to allow service restart with an option to recreate containers, addressing WSL2 bind mount issues.
- Added new chat configuration management functions in `system_controller.py` for loading, saving, and validating chat prompts.
- Introduced `ChatSettings` component in the web UI for admin users to manage chat configurations easily.
- Updated API service methods in `api.ts` to support chat configuration endpoints.
- Integrated chat settings into the system management page for better accessibility.
* Refactor backend shutdown process and enhance chat service configuration logging
- Updated `start.sh` to improve shutdown handling by explicitly killing the backend process if running.
- Modified `chat_service.py` to enhance logging for loading chat system prompts, providing clearer feedback on configuration usage.
- Added a new `chat` field in `model_registry.py` for better chat service configuration management.
- Updated vector store query parameters in `vector_stores.py` for improved clarity and functionality.
- Enhanced the chat component in the web UI to conditionally auto-scroll based on message sending status.
* Implement plugin system for enhanced functionality and configuration management
- Introduced a new plugin architecture to allow for extensibility in the Chronicle application.
- Added Home Assistant plugin for controlling devices via natural language commands triggered by wake words.
- Implemented plugin configuration management endpoints in the API for loading, saving, and validating plugin settings.
- Enhanced the web UI with a dedicated Plugins page for managing plugin configurations.
- Updated Docker Compose files to include Tailscale integration for remote service access.
- Refactored existing services to support plugin interactions during conversation and memory processing.
- Improved error handling and logging for plugin initialization and execution processes.
* Enhance configuration management and plugin system integration
- Updated .gitignore to include plugins.yml for security reasons.
- Modified start.sh to allow passing additional arguments during service startup.
- Refactored wizard.py to support new HF_TOKEN configuration prompts and improved handling of wake words in plugin settings.
- Introduced a new setup_hf_token_if_needed function to streamline Hugging Face token management.
- Enhanced the GitHub Actions workflow to create plugins.yml from a template, ensuring proper configuration setup.
- Added detailed comments and documentation in the plugins.yml.template for better user guidance on Home Assistant integration.
* Implement Redis integration for client-user mapping and enhance wake word processing
- Added asynchronous Redis support in ClientManager for tracking client-user relationships.
- Introduced `initialize_redis_for_client_manager` to set up Redis for cross-container mapping.
- Updated `create_client_state` to use asynchronous tracking for client-user relationships.
- Enhanced wake word processing in PluginRouter with normalization and command extraction.
- Refactored DeepgramStreamingConsumer to utilize async Redis lookups for user ID retrieval.
- Set TTL on Redis streams during client state cleanup for better resource management.
* Refactor Deepgram worker management and enhance text normalization
- Disabled the batch Deepgram worker in favor of the streaming worker to prevent race conditions.
- Updated text normalization in wake word processing to replace punctuation with spaces, preserving word boundaries.
- Enhanced regex pattern for wake word matching to allow optional punctuation and whitespace after the last part.
- Improved logging in DeepgramStreamingConsumer for better visibility of message processing and error handling.
* Add original prompt retrieval and restoration in chat configuration test
- Implemented retrieval of the original chat prompt before saving a custom prompt to ensure test isolation.
- Added restoration of the original prompt after the test to prevent interference with subsequent tests.
- Enhanced the test documentation for clarity on the purpose of these changes.
* Refactor test execution and enhance documentation for integration tests
- Simplified test execution commands in CLAUDE.md and quickstart.md for better usability.
- Added instructions for running tests from the project root and clarified the process for executing the complete Robot Framework test suite.
- Introduced a new Docker service for the Deepgram streaming worker in docker-compose-test.yml to improve testing capabilities.
- Updated system_admin_tests.robot to use a defined default prompt for restoration, enhancing test reliability and clarity.
* Enhance test environment cleanup and improve Deepgram worker management
- Updated `run-test.sh` and `run-robot-tests.sh` to improve cleanup processes, including handling permission issues with Docker.
- Introduced a new function `mark_session_complete` in `session_controller.py` to ensure atomic updates for session completion status.
- Refactored WebSocket and conversation job handling to utilize the new session completion function, enhancing reliability.
- Updated `start-workers.sh` to enable the batch Deepgram worker alongside the streaming worker for improved transcription capabilities.
- Enhanced test scripts to verify the status of Deepgram workers and ensure proper cleanup of test containers.
* Refactor worker management and introduce orchestrator for improved process handling
- Replaced the bash-based `start-workers.sh` script with a Python-based worker orchestrator for better process management and health monitoring.
- Updated `docker-compose.yml` to configure the new orchestrator and adjust worker definitions, including the addition of audio persistence and stream workers.
- Enhanced the Dockerfile to remove the old startup script and ensure the orchestrator is executable.
- Introduced new modules for orchestrator configuration, health monitoring, process management, and worker registry to streamline worker lifecycle management.
- Improved environment variable handling for worker configuration and health checks.
* oops
* oops2
* Remove legacy test runner script and update worker orchestration
- Deleted the `run-test.sh` script, which was used for local test execution.
- Updated Docker configurations to replace the `start-workers.sh` script with `worker_orchestrator.py` for improved worker management.
- Enhanced health monitoring and process management in the orchestrator to ensure better reliability and logging.
- Adjusted deployment configurations to reflect the new orchestrator setup.
* Add bulk restart mechanism for RQ worker registration loss
- Introduced a new method `_handle_registration_loss` to manage RQ worker registration loss, replicating the behavior of the previous bash script.
- Implemented a cooldown period to prevent frequent restarts during network issues.
- Added logging for bulk restart actions and their outcomes to enhance monitoring and debugging capabilities.
- Created a `_restart_all_rq_workers` method to facilitate the bulk restart of RQ workers, ensuring they re-register with Redis upon startup.
* Enhance plugin architecture with event-driven system and test integration
- Introduced a new Test Event Plugin to log all plugin events to an SQLite database for integration testing.
- Updated the plugin system to utilize event subscriptions instead of access levels, allowing for more flexible event handling.
- Refactored the PluginRouter to dispatch events based on subscriptions, improving the event-driven architecture.
- Enhanced Docker configurations to support development and testing environments with appropriate dependencies.
- Added comprehensive integration tests to verify the functionality of the event dispatch system and plugin interactions.
- Updated documentation and test configurations to reflect the new event-based plugin structure.
* Enhance Docker configurations and startup script for test mode
- Updated `docker-compose-test.yml` to include a test command for services, enabling a dedicated test mode.
- Modified `start.sh` to support a `--test` flag, allowing the FastAPI backend to run with test-specific configurations.
- Adjusted worker commands to utilize the `--group test` option in test mode for improved orchestration and management.
* Refactor test scripts for improved reliability and clarity
- Updated `run-robot-tests.sh` to enhance the verification of the Deepgram batch worker process, ensuring non-numeric characters are removed from the check.
- Modified `plugin_tests.robot` to use a more explicit method for checking the length of subscriptions and added a skip condition for unavailable audio files.
- Adjusted `plugin_event_tests.robot` to load the test audio file from a variable, improving test data management.
- Refactored `plugin_keywords.robot` to utilize clearer length checks for subscriptions and event parts, enhancing readability and maintainability.
* remove mistral deadcode; notebooks untouched
* Refactor audio streaming endpoints and improve documentation
- Updated WebSocket endpoints to use a unified format with codec parameters (`/ws?codec=pcm` and `/ws?codec=opus`) for audio streaming, replacing the previous `/ws_pcm` and `/ws_omi` endpoints.
- Enhanced documentation to reflect the new endpoint structure and clarify audio processing capabilities.
- Removed deprecated audio cropping functionality and related configurations to streamline the audio processing workflow.
- Updated various components and scripts to align with the new endpoint structure, ensuring consistent usage across the application.
* Enhance testing infrastructure and API routes for plugin events
- Updated `docker-compose-test.yml` to introduce low speech detection thresholds for testing, improving the accuracy of speech detection during tests.
- Added new test-only API routes in `test_routes.py` for clearing and retrieving plugin events, ensuring a clean state between tests.
- Refactored existing test scripts to utilize the new API endpoints for event management, enhancing test reliability and clarity.
- Improved logging and error handling in various components to facilitate debugging during test execution.
- Adjusted environment variable handling in test setup scripts to streamline configuration and improve flexibility.
* Add audio pipeline architecture documentation and improve audio persistence worker configuration
- Introduced a comprehensive documentation file detailing the audio pipeline architecture, covering data flow, processing stages, and key components.
- Enhanced the audio persistence worker setup by implementing multiple concurrent workers to improve audio processing efficiency.
- Adjusted sleep intervals in the audio streaming persistence job for better responsiveness and event loop yielding.
- Updated test script to run the full suite of integration tests from the specified directory, ensuring thorough testing coverage.
* Add test container setup and teardown scripts
- Introduced `setup-test-containers.sh` for streamlined startup of test containers, including health checks and environment variable loading.
- Added `teardown-test-containers.sh` for simplified container shutdown, with options to remove volumes.
- Enhanced user feedback with color-coded messages for better visibility during test setup and teardown processes.
* Update worker count validation and websocket disconnect tests
- Adjusted worker count expectations in the Worker Count Validation Test to reflect an increase from 7 to 9 workers, accounting for additional audio persistence workers.
- Enhanced the WebSocket Disconnect Conversation End Reason Test by adding steps to maintain audio streaming during disconnection, ensuring accurate simulation of network dropout scenarios.
- Improved comments for clarity and added critical notes regarding inactivity timeout handling.
* Refactor audio storage to MongoDB chunks and enhance cleanup settings management
- Replaced the legacy AudioFile model with AudioChunkDocument for storing audio data in MongoDB, optimizing storage and retrieval.
- Introduced CleanupSettings dataclass for managing soft-deletion configurations, including auto-cleanup and retention days.
- Added admin API routes for retrieving and saving cleanup settings, ensuring better control over data retention policies.
- Updated audio processing workflows to utilize MongoDB chunks, removing dependencies on disk-based audio files.
- Enhanced tests to validate the new audio chunk storage and cleanup functionalities, ensuring robust integration with existing systems.
* Refactor audio processing to utilize MongoDB chunks and enhance job handling
- Removed audio file path parameters from various functions, transitioning to audio data retrieval from MongoDB chunks.
- Updated the `start_post_conversation_jobs` function to reflect changes in audio handling, ensuring jobs reconstruct audio from database chunks.
- Enhanced the `transcribe_full_audio_job` and `recognise_speakers_job` to process audio directly from memory, eliminating the need for temporary files.
- Improved error handling and logging for audio data retrieval, ensuring better feedback during processing.
- Added a new utility function for converting PCM data to WAV format in memory, streamlining audio format handling.
* Refactor speaker recognition client to use in-memory audio data
- Updated methods to accept audio data as bytes instead of file paths, enhancing performance by eliminating disk I/O.
- Improved logging to reflect in-memory audio processing, providing better insights during speaker identification and diarization.
- Streamlined audio data handling in the `diarize_identify_match` and `diarize_and_identify` methods, ensuring consistency across the client.
- Removed temporary file handling, simplifying the audio processing workflow and reducing potential file system errors.
* Add mock providers and update testing workflows for API-independent execution
- Introduced `MockLLMProvider` and `MockTranscriptionProvider` to facilitate testing without external API dependencies, allowing for consistent and controlled test environments.
- Created `run-no-api-tests.sh` script to execute tests that do not require API keys, ensuring separation of API-dependent and independent tests.
- Updated Robot Framework test configurations to utilize mock services, enhancing test reliability and reducing external dependencies.
- Modified existing test workflows to include new configurations and ensure proper handling of results for tests excluding API keys.
- Added `mock-services.yml` configuration to disable external API services while maintaining core functionality for testing purposes.
- Enhanced documentation to reflect the new tagging system for tests requiring API keys, improving clarity on test execution requirements.
* Enhance testing documentation and workflows for API key separation
- Updated CLAUDE.md to clarify test execution modes, emphasizing the separation of tests requiring API keys from those that do not.
- Expanded the testing guidelines in TESTING_GUIDELINES.md to detail the organization of tests based on API dependencies, including tagging conventions and execution paths.
- Improved mock-services.yml to include dummy configurations for LLM and embedding services, ensuring tests can run without actual API calls.
- Added comprehensive documentation on GitHub workflows for different test scenarios, enhancing clarity for contributors and maintainers.
* Update test configurations and documentation for API key management
- Modified `plugins.yml.template` to implement event subscriptions for the Home Assistant plugin, enhancing its event-driven capabilities.
- Revised `README.md` to clarify test execution processes, emphasizing the distinction between tests requiring API keys and those that do not.
- Updated `mock-services.yml` to streamline mock configurations, ensuring compatibility with the new testing workflows.
- Added `requires-api-keys` tags to relevant test cases across various test files, improving organization and clarity regarding API dependencies.
- Enhanced documentation for test scripts and configurations, providing clearer guidance for contributors on executing tests based on API key requirements.
* Add optional service profile to Docker Compose test configuration
* Refactor audio processing and job handling for transcription workflows
- Updated `upload_and_process_audio_files` and `start_post_conversation_jobs` to enqueue transcription jobs separately for file uploads, ensuring accurate processing order.
- Enhanced logging to provide clearer insights into job enqueuing and processing stages.
- Removed batch transcription from the post-conversation job chain for streaming audio, utilizing the streaming transcript directly.
- Introduced word-level timestamps in the `Conversation` model to improve transcript detail and accuracy.
- Updated tests to reflect changes in job handling and ensure proper verification of post-conversation processing.
* Remove unnecessary network aliases from speaker service in Docker Compose configuration
* Add network aliases for speaker service in Docker Compose configuration
* Refactor Conversation model to use string for provider field
- Updated the `Conversation` model to replace the `TranscriptProvider` enum with a string type for the `provider` field, allowing for greater flexibility in provider names.
- Adjusted related job functions to accommodate this change, simplifying provider handling in the transcription workflow.
* Enhance configuration and model handling for waveform data
- Updated Docker Compose files to mount the entire config directory, allowing for better management of configuration files.
- Introduced a new `WaveformData` model to store pre-computed waveform visualization data, improving UI performance by enabling waveform display without real-time decoding.
- Enhanced the `app_factory` and `job` models to include the new `WaveformData` model, ensuring proper initialization and data handling.
- Implemented waveform generation logic in a new worker module, allowing for on-demand waveform creation from audio chunks.
- Added API endpoints for retrieving and generating waveform data, improving the overall audio processing capabilities.
- Updated tests to cover new functionality and ensure robustness in waveform data handling.
* Add SDK testing scripts for authentication, conversation retrieval, and audio upload
- Introduced three new test scripts: `sdk_test_auth.py`, `sdk_test_conversations.py`, and `sdk_test_upload.py`.
- Each script tests different functionalities of the SDK, including authentication, conversation retrieval, and audio file uploads.
- The scripts utilize the `ChronicleClient` to perform operations and print results for verification.
- Enhanced testing capabilities for the SDK, ensuring robust validation of core features.
* Enhance audio processing and conversation handling for large files
- Added configuration options for speaker recognition chunking in `.env.template`, allowing for better management of large audio files.
- Updated `get_conversations` function to include an `include_deleted` parameter for filtering conversations based on their deletion status.
- Enhanced `finalize_session` method in `AudioStreamProducer` to send an end marker to Redis, ensuring proper session closure.
- Introduced `reconstruct_audio_segments` function to yield audio segments with overlap for efficient processing of lengthy conversations.
- Implemented merging of overlapping speaker segments to improve accuracy in speaker recognition.
- Added integration tests for WebSocket streaming transcription to validate the end_marker functionality and overall transcription flow.
* archive
* Implement annotation system and enhance audio processing capabilities
- Introduced a new annotation model to support user edits and AI-powered suggestions for memories and transcripts.
- Added annotation routes for CRUD operations, enabling the creation and management of annotations via the API.
- Enhanced the audio processing workflow to support fetching audio segments from the backend, improving speaker recognition accuracy.
- Updated the speaker recognition client to handle conversation-based audio fetching, allowing for better management of large audio files.
- Implemented a cron job for generating AI suggestions on potential errors in transcripts and memories, improving user experience and content accuracy.
- Enhanced the web UI to support inline editing of transcript segments and memory content, providing a more interactive user experience.
- Updated configuration files to support new features and improve overall system flexibility.
* Implement OmegaConf-based configuration management for backend settings
- Introduced a new configuration loader using OmegaConf for unified management of backend settings.
- Updated existing configuration functions to leverage the new loader, enhancing flexibility and maintainability.
- Added support for environment variable interpolation in configuration files.
- Refactored various components to retrieve settings from the new configuration system, improving consistency across the application.
- Updated requirements to include OmegaConf as a dependency.
- Enhanced documentation and comments for clarity on configuration management.
* Refactor .env.template and remove unused diarization configuration
- Updated the .env.template to clarify its purpose for secret values and streamline setup instructions.
- Removed the deprecated diarization_config.json.template file, as it is no longer needed.
- Added new environment variables for Langfuse and Tailscale integration to enhance observability and remote service access.
* Implement legacy environment variable syntax support in configuration loader
- Added custom OmegaConf resolvers to handle legacy ${VAR:-default} syntax for backward compatibility.
- Introduced a preprocessing function to convert legacy syntax in YAML files to OmegaConf-compatible format.
- Updated the load_config function to utilize the new preprocessing for loading defaults and user configurations.
- Enhanced documentation for clarity on the new legacy syntax handling.
* Add plugins configuration path retrieval and refactor usage
- Introduced a new function `get_plugins_yml_path` to centralize the retrieval of the plugins.yml file path.
- Updated `system_controller.py` and `plugin_service.py` to use the new function for improved maintainability and consistency in accessing the plugins configuration.
- Enhanced code clarity by removing hardcoded paths and utilizing the centralized configuration method.
* Unify plugin terminology and fix memory job dependencies
Plugin terminology: subscriptions→events, trigger→condition
Memory jobs: no longer blocked by disabled speaker recognition
* Update Docker Compose configuration and enhance system routes
- Updated Docker Compose files to mount the entire config directory, consolidating configuration management.
- Refactored the `save_diarization_settings` function to improve clarity and maintainability by renaming it to `save_diarization_settings_controller`.
- Enhanced the System component in the web UI to include configuration diagnostics, providing better visibility into system health and issues.
* circular import
* Refactor testing infrastructure and enhance container management
- Updated the testing documentation to reflect a new Makefile-based approach for running tests and managing containers.
- Introduced new scripts for container management, including starting, stopping, restarting, and cleaning containers while preserving logs.
- Added a cleanup script to handle data ownership and permissions correctly.
- Implemented a logging system that saves container logs automatically before cleanup.
- Enhanced the README with detailed instructions for running tests and managing the test environment.
* Add Email Summarizer Plugin and SMTP Email Service
- Introduced the Email Summarizer Plugin that automatically sends email summaries upon conversation completion.
- Implemented SMTP Email Service for sending emails, supporting HTML and plain text formats with TLS/SSL encryption.
- Added configuration options for SMTP settings in the .env.template and plugins.yml.template.
- Created comprehensive documentation for plugin development and usage, including a new plugin generation script.
- Enhanced testing coverage for the Email Summarizer Plugin and SMTP Email Service to ensure reliability and functionality.
* Refactor plugin management and introduce Email Summarizer setup
- Removed the static PLUGINS dictionary and replaced it with a dynamic discovery mechanism for plugins.
- Implemented a new setup process for plugins, allowing for configuration via individual setup scripts.
- Added the Email Summarizer plugin with a dedicated setup script for SMTP configuration.
- Enhanced the main setup flow to support community plugins and their configuration.
- Cleaned up unused functions related to plugin configuration and streamlined the overall plugin setup process.
* Enhance plugin configuration and documentation
- Updated the .env.template to include new configuration options for the Home Assistant and Email Summarizer plugins, including server URLs, tokens, and additional settings.
- Refactored Docker Compose files to correctly mount plugin configuration paths.
- Introduced comprehensive documentation for plugin configuration architecture, detailing the separation of concerns for orchestration, settings, and secrets.
- Added individual configuration files for the Home Assistant and Email Summarizer plugins, ensuring proper management of non-secret settings and environment variable references.
- Improved the plugin loading process to merge configurations from multiple sources, enhancing flexibility and maintainability.
* Refactor plugin setup process to allow interactive user input
- Updated the plugin setup script to run interactively, enabling plugins to prompt for user input during configuration.
- Removed output capturing to facilitate real-time interaction and improved error messaging to include exit codes for better debugging.
* Add shared setup utilities for interactive configuration
- Introduced `setup_utils.py` containing functions for reading environment variables, prompting user input, and masking sensitive values.
- Refactored existing code in `wizard.py` and `init.py` to utilize these shared utilities, improving code reuse and maintainability.
- Updated documentation to include usage examples for the new utilities in plugin setup scripts, enhancing developer experience and clarity.
* Enhance plugin security architecture and configuration management
- Introduced a three-file separation for plugin configuration to improve security:
- `backends/advanced/.env` for secrets (gitignored)
- `config/plugins.yml` for orchestration with environment variable references
- `plugins/{plugin_id}/config.yml` for non-secret defaults
- Updated documentation to emphasize the importance of using `${ENV_VAR}` syntax for sensitive data and provided examples of correct usage.
- Enhanced the Email Summarizer plugin setup process to automatically update `config/plugins.yml` with environment variable references, ensuring secrets are not hardcoded.
- Added new fields to the User model for notification email management and improved error logging in user-related functions.
- Refactored audio chunk utilities to use a consistent method for fetching conversation metadata.
* Refactor backend components for improved functionality and stability
- Added a new parameter `transcript_version_id` to the `open_conversation_job` function to support streaming transcript versioning.
- Enhanced error handling in `check_enrolled_speakers_job` and `recognise_speakers_job` to allow conversations to proceed even when the speaker service is unavailable, improving resilience.
- Updated `send_to_adv.py` to support dynamic WebSocket and HTTP protocols based on environment settings, enhancing configuration flexibility.
- Introduced a background task in `send_to_adv.py` to handle incoming messages from the backend, ensuring connection stability and logging interim results.
* Refactor plugin setup timing to enhance configuration flow
* Refactor save_diarization_settings_controller to improve validation and error handling
- Updated the controller to filter out invalid settings instead of raising an error for each unknown key, allowing for more flexible input.
- Added a check to reject requests with no valid settings provided, enhancing robustness.
- Adjusted logging to reflect the filtered settings being saved.
* Refactor audio processing and conversation management for improved deduplication and tracking
* Refactor audio and email handling for improved functionality and security
- Updated `mask_value` function to handle whitespace more effectively.
- Enhanced `create_plugin` to remove existing directories when using the `--force` option.
- Changed logging level from error to debug for existing admin user checks.
- Improved client ID generation logging for clarity.
- Removed unused fields from conversation creation.
- Added HTML escaping in email templates to prevent XSS attacks.
- Updated audio file download function to include user ID for better tracking.
- Adjusted WebSocket connection settings to respect SSL verification based on environment variables.
* Refactor audio upload functionality to remove unused parameters
- Removed `auto_generate_client` and `folder` parameters from audio upload functions to streamline the API.
- Updated related function calls and documentation to reflect these changes, enhancing clarity and reducing complexity.
* Refactor Email Summarizer plugin configuration for improved clarity and security
- Removed outdated migration instructions from `plugin-configuration.md` to streamline documentation.
- Enhanced `README.md` to clearly outline the three-file separation for plugin configuration, emphasizing the roles of `.env`, `config.yml`, and `plugins.yml`.
- Updated `setup.py` to reflect changes in orchestration settings, ensuring only relevant configurations are included in `config/plugins.yml`.
- Improved security messaging to highlight the importance of not committing secrets to version control.
* Update API key configuration in config.yml.template to use environment variable syntax for improved flexibility and security. This change standardizes the way API keys are referenced across different models and services. (#273)
Co-authored-by: roshan.john <roshanjohn1460@gmail.com>
* Refactor Redis job queue cleanup process for improved success tracking
- Replaced total job count with separate counters for successful and failed jobs during Redis queue cleanup.
- Enhanced logging to provide detailed feedback on the number of jobs cleared and any failures encountered.
- Improved error handling to ensure job counts are accurately reflected even when exceptions occur.
* fix tests
* Update CI workflows to use 'docker compose' for log retrieval and added container status check
- Replaced 'docker logs' commands with 'docker compose -f docker-compose-test.yml logs' for consistency across workflows.
- Added a check for running containers before saving logs to enhance debugging capabilities.
* test fixes
* FIX StreamingTranscriptionConsumer to support cumulative audio timestamp adjustments
- Added `audio_offset_seconds` to track cumulative audio duration for accurate timestamp adjustments across transcription sessions.
- Updated `store_final_result` method to adjust word and segment timestamps based on cumulative audio offset.
- Improved logging to reflect changes in audio offset after storing results.
- Modified Makefile and documentation to clarify test execution options, including new tags for slow and SDK tests, enhancing test organization and execution clarity.
* Enhance test container setup and improve error messages in integration tests
- Set `COMPOSE_PROJECT_NAME` for test containers to ensure consistent naming.
- Consolidated error messages in the `websocket_transcription_e2e_test.robot` file for clarity, improving readability and debugging.
* Improve WebSocket closing logic and enhance integration test teardown
- Added timeout handling for WebSocket closure in `AudioStreamClient` to prevent hanging and ensure clean disconnection.
- Updated integration tests to log the total chunks sent when closing audio streams, improving clarity on resource management during test teardown.
* Refactor job status handling to align with RQ standards
- Updated job status checks across various modules to use "started" and "finished" instead of "processing" and "completed" for consistency with RQ's naming conventions.
- Adjusted related logging and response messages to reflect the new status terminology.
- Simplified Docker Compose project name handling in test scripts to avoid conflicts and improve clarity in test environment setup.
* Update test configurations and improve audio inactivity handling
- Increased `SPEECH_INACTIVITY_THRESHOLD_SECONDS` to 20 seconds in `docker-compose-test.yml` for better audio duration handling during tests.
- Refactored session handling in `session_controller.py` to clarify client ID usage.
- Updated `conversation_utils.py` to track speech activity using audio timestamps, enhancing accuracy in inactivity detection.
- Simplified test scripts by removing unnecessary `COMPOSE_PROJECT_NAME` references, aligning with the new project naming convention.
- Adjusted integration tests to reflect changes in inactivity timeout and ensure proper handling of audio timestamps.
* Refactor audio processing and enhance error handling
- Updated `worker_orchestrator.py` to use `logger.exception` for improved error logging.
- Changed default MongoDB database name from "friend-lite" to "chronicle" in multiple files for consistency.
- Added a new method `close_stream_without_stop` in `audio_stream_client.py` to handle abrupt WebSocket disconnections.
- Enhanced audio validation in `audio_utils.py` to support automatic resampling of audio data if sample rates do not match.
- Improved logging in various modules to provide clearer insights during audio processing and event dispatching.
* Enhance Docker command handling and configuration management
- Updated `run_compose_command` to support separate build commands for services, including profile management for backend and speaker-recognition services.
- Improved error handling and output streaming during Docker command execution.
- Added `ensure_docker_network` function to verify and create the required Docker network before starting services.
- Refactored configuration files to utilize `oc.env` for environment variable management, ensuring better compatibility and flexibility across different environments.
* Enhance configuration loading to support custom config file paths
- Added support for the CONFIG_FILE environment variable to allow specifying custom configuration files for testing.
- Implemented logic to handle both absolute paths and relative filenames for the configuration file, improving flexibility in configuration management.
* Update test scripts to use TEST_CONFIG_FILE for configuration management
- Replaced CONFIG_FILE with TEST_CONFIG_FILE in both run-no-api-tests.sh and run-robot-tests.sh to standardize configuration file usage.
- Updated paths to point to mock and deepgram-openai configuration files inside the container, improving clarity and consistency in test setups.
* Refactor audio upload response handling and improve error reporting
- Updated `upload_and_process_audio_files` to return appropriate HTTP status codes based on upload results: 400 for all failures, 207 for partial successes, and 200 for complete success.
- Enhanced error messages in the audio upload tests to provide clearer feedback on upload failures, including specific error details for better debugging.
- Adjusted test scripts to ensure consistent handling of conversation IDs in job metadata, improving validation checks for job creation.
* Refactor audio processing and job handling to improve transcription management
- Updated `upload_and_process_audio_files` to check for transcription provider availability before enqueueing jobs, enhancing error handling and logging.
- Modified `start_post_conversation_jobs` to conditionally enqueue memory extraction jobs based on configuration, improving flexibility in job management.
- Enhanced event dispatch job dependencies to only include jobs that were actually enqueued, ensuring accurate job tracking.
- Added `is_transcription_available` function to check transcription provider status, improving modularity and clarity in the transcription workflow.
* Enhance integration tests for plugin events and improve error handling
- Updated integration tests to filter plugin events by conversation ID, ensuring accurate event tracking and reducing noise from fixture events.
- Improved error messages in event verification to include conversation ID context, enhancing clarity during test failures.
- Refactored audio upload handling to check for transcription job creation, allowing for more robust conversation polling and error reporting.
- Added new keyword to verify conversation end reasons, improving test coverage for conversation state validation.
* Enhance speaker recognition testing and audio processing
- Added mock speaker recognition client to facilitate testing without resource-intensive dependencies.
- Updated Docker Compose configurations to include mock speaker client for test environments.
- Refactored audio segment reconstruction to ensure precise clipping based on time boundaries.
- Improved error handling in transcription jobs and speaker recognition workflows to enhance robustness.
- Adjusted integration tests to utilize real-time pacing for audio chunk streaming, improving test accuracy.
* Refactor audio chunk retrieval and enhance logging in audio processing
- Introduced logging for audio chunk requests to improve traceability.
- Replaced manual audio chunk processing with a dedicated `reconstruct_audio_segment` function for better clarity and efficiency.
- Improved error handling during audio reconstruction to provide more informative responses in case of failures.
- Cleaned up imports and removed redundant code related to audio chunk calculations.
* Refactor mock speaker recognition client and improve testing structure
- Replaced direct import of mock client with a structured import from the new testing module.
- Introduced a dedicated `mock_speaker_client.py` to provide a mock implementation for speaker recognition, facilitating testing without heavy dependencies.
- Added an `__init__.py` file in the testing directory to organize testing utilities and mocks.
* Enhance conversation model to include word-level timestamps and improve transcript handling
- Added a new `words` field to the `Conversation` model for storing word-level timestamps.
- Updated methods to handle word data during transcript version creation, ensuring compatibility with speaker recognition.
- Refactored conversation job processing to utilize the new word structure, improving data integrity and access.
- Enhanced speaker recognition job to read words from the new standardized location, ensuring backward compatibility with legacy data.
* Implement speaker reprocessing feature and enhance timeout calculation
- Added a new endpoint to reprocess speaker identification for existing transcripts, creating a new version with re-identified speakers.
- Introduced a method to calculate proportional t…
AnkushMalaker
added a commit
that referenced
this pull request
Feb 19, 2026
* audio upload extension with gdrive credentials
* FIX: API parameters
* UPDATE: tmp files cleanup n code refactored as per review
* REFACTOR: minor refactor as per review
* REFACTOR: minor update as per review
* UPDATE: gdrive sync logic
* REFACTOR: code update as per gdrive and update credential client
* REFACTOR: validation updated - as per review from CR
* UPDATE: code has been refactore for UUID for diffrent audio upload sources
* REFACTOR: updated code as per review
* Update documentation and configuration to reflect the transition from 'friend-backend' to 'chronicle-backend' across various files, including setup instructions, Docker configurations, and service logs.
* Update test script to use docker-compose-test.yml for all test-related operations
* Added standard MIT license
* Fix/cleanup model (#219)
* refactor memory
* add config
* docstring
* more cleanup
* code quality
* code quality
* unused return
* DOTTED GET
* Refactor Docker and CI configurations
- Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process.
- Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services.
- Added new dependencies for Google API clients in `uv.lock` to support upcoming features.
* Update configuration files for model providers and Docker setup
- Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram.
- Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving.
- Updated memory configuration endpoint to accept plain text for YAML input.
* Update transcription job handling to format speaker IDs
- Changed variable name from `speaker_name` to `speaker_id` for clarity.
- Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling.
* Remove loading of backend .env file in test environment setup
- Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests.
* Enhance configuration management and setup wizard
- Updated README to reflect the new setup wizard process.
- Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers.
- Improved user feedback during configuration updates, including success messages for configuration file updates.
- Enabled backup of existing `config.yml` before saving changes.
* Enhance HTTPS configuration in setup wizard
- Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates.
- Improved user prompts for server IP/domain input during HTTPS setup.
- Updated default behavior to use existing IP or localhost based on user input.
- Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access.
* Add source parameter to audio file writing in websocket controller
- Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking.
---------
Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
* fix/broken-tests (#230)
* refactor memory
* add config
* docstring
* more cleanup
* code quality
* code quality
* unused return
* DOTTED GET
* Refactor Docker and CI configurations
- Removed the creation of `memory_config.yaml` from the CI workflow to streamline the process.
- Updated Docker Compose files to mount `config.yml` for model registry and memory settings in both services.
- Added new dependencies for Google API clients in `uv.lock` to support upcoming features.
* Update configuration files for model providers and Docker setup
- Changed LLM, embedding, and STT providers in `config.yml` to OpenAI and Deepgram.
- Removed read-only flag from `config.yml` in Docker Compose files to allow UI configuration saving.
- Updated memory configuration endpoint to accept plain text for YAML input.
* Update transcription job handling to format speaker IDs
- Changed variable name from `speaker_name` to `speaker_id` for clarity.
- Added logic to convert integer speaker IDs from Deepgram to string format for consistent speaker labeling.
* Remove loading of backend .env file in test environment setup
- Eliminated the code that loads the .env file from the backends/advanced directory, simplifying the environment configuration for tests.
* Enhance configuration management and setup wizard
- Updated README to reflect the new setup wizard process.
- Added functionality to load and save `config.yml` in the setup wizard, including default configurations for LLM and memory providers.
- Improved user feedback during configuration updates, including success messages for configuration file updates.
- Enabled backup of existing `config.yml` before saving changes.
* Enhance HTTPS configuration in setup wizard
- Added functionality to check for existing SERVER_IP in the environment file and prompt the user to reuse or enter a new IP for SSL certificates.
- Improved user prompts for server IP/domain input during HTTPS setup.
- Updated default behavior to use existing IP or localhost based on user input.
- Changed RECORD_ONLY_ENROLLED_SPEAKERS setting in the .env template to false for broader access.
* Add source parameter to audio file writing in websocket controller
- Included a new `source` parameter with the value "websocket" in the `_process_batch_audio_complete` function to enhance audio file context tracking.
* Refactor error handling in system controller and update memory config routes
- Replaced ValueError with HTTPException for better error handling in `save_diarization_settings` and `validate_memory_config` functions.
- Introduced a new Pydantic model, `MemoryConfigRequest`, for validating memory configuration requests in the system routes.
- Updated the `validate_memory_config` endpoint to accept the new request model, improving input handling and validation.
---------
Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
* Feat/add obsidian 3 (#233)
* obsidian support
* neo4j comment
* cleanup code
* unused line
* unused line
* Fix MemoryEntry object usage in chat service
* comment
* feat(obsidian): add obsidian memory search integration to chat
* unit test
* use rq
* neo4j service
* typefix
* test fix
* cleanup
* cleanup
* version changes
* profile
* remove unused imports
* Refactor memory configuration validation endpoints
- Removed the deprecated `validate_memory_config_raw` endpoint and replaced it with a new endpoint that accepts plain text for validation.
- Updated the existing `validate_memory_config` endpoint to clarify that it now accepts JSON input.
- Adjusted the API call in the frontend to point to the new validation endpoint.
* Refactor health check model configuration loading
- Updated the health check function to load model configuration from the models registry instead of the root config.
- Improved error handling by logging warnings when model configuration loading fails.
---------
Co-authored-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
* Update .gitignore to exclude all files in app/ios and app/android directories (#238)
* fix: Copy full source code in speaker-recognition Dockerfile (#243)
Adds COPY src/ src/ step after dependency installation to ensure
all source files are available in the Docker image. This improves
build caching while ensuring complete source code is present.
* Enhance configuration management and add new setup scripts (#235)
* Enhance configuration management and add new setup scripts
- Updated .gitignore to include config.yml and its template.
- Added config.yml.template for default configuration settings.
- Introduced restart.sh script for service management.
- Enhanced services.py to load config.yml and check for Obsidian/Neo4j integration.
- Updated wizard.py to prompt for Obsidian/Neo4j configuration during setup and create config.yml from template if it doesn't exist.
* Refactor transcription providers and enhance configuration management
- Updated Docker Compose files to include the new Neo4j service configuration.
- Added support for Obsidian/Neo4j integration in the setup process.
- Refactored transcription providers to utilize a registry-driven approach for Deepgram and Parakeet.
- Enhanced error handling and logging in transcription processes.
- Improved environment variable management in test scripts to prioritize command-line overrides.
- Removed deprecated Parakeet provider implementation and streamlined audio stream workers.
* Update configuration management and enhance file structure, add test-matrix (#237)
* Update configuration management and enhance file structure
- Refactored configuration file paths to use a dedicated `config/` directory, including updates to `config.yml` and its template.
- Modified service scripts to load the new configuration path for `config.yml`.
- Enhanced `.gitignore` to include the new configuration files and templates.
- Updated documentation to reflect changes in configuration file locations and usage.
- Improved setup scripts to ensure proper creation and management of configuration files.
- Added new test configurations for various provider combinations to streamline testing processes.
* Add test requirements and clean up imports in wizard.py
- Introduced a new `test-requirements.txt` file to manage testing dependencies.
- Removed redundant import of `shutil` in `wizard.py` to improve code clarity.
* Add ConfigManager for unified configuration management
- Introduced a new `config_manager.py` module to handle reading and writing configurations from `config.yml` and `.env` files, ensuring backward compatibility.
- Refactored `ChronicleSetup` in `backends/advanced/init.py` to utilize `ConfigManager` for loading and updating configurations, simplifying the setup process.
- Removed redundant methods for loading and saving `config.yml` directly in `ChronicleSetup`, as these are now managed by `ConfigManager`.
- Enhanced user feedback during configuration updates, including success messages for changes made to configuration files.
* Refactor transcription provider configuration and enhance setup process
- Updated `.env.template` to clarify speech-to-text configuration and removed deprecated options for Mistral.
- Modified `docker-compose.yml` to streamline environment variable management by removing unused Mistral keys.
- Enhanced `ChronicleSetup` in `init.py` to provide clearer user feedback and updated the transcription provider selection process to rely on `config.yml`.
- Improved error handling in the websocket controller to determine the transcription provider from the model registry instead of environment variables.
- Updated health check routes to reflect the new method of retrieving the transcription provider from `config.yml`.
- Adjusted `config.yml.template` to include comments on transcription provider options for better user guidance.
* Enhance ConfigManager with deep merge functionality
- Updated the `update_memory_config` method to perform a deep merge of updates into the memory configuration, ensuring nested dictionaries are merged correctly.
- Added a new `_deep_merge` method to handle recursive merging of dictionaries, improving configuration management capabilities.
* Refactor run-test.sh and enhance memory extraction tests
- Removed deprecated environment variable handling for TRANSCRIPTION_PROVIDER in `run-test.sh`, streamlining the configuration process.
- Introduced a new `run-custom.sh` script for executing Robot tests with custom configurations, improving test flexibility.
- Enhanced memory extraction tests in `audio_keywords.robot` and `memory_keywords.robot` to include detailed assertions and result handling.
- Updated `queue_keywords.robot` to fail fast if a job is in a 'failed' state when expecting 'completed', improving error handling.
- Refactored `test_env.py` to load environment variables with correct precedence, ensuring better configuration management.
* unify tests to robot test, add some more clean up
* Update health check configuration in docker-compose-test.yml (#241)
- Increased the number of retries from 5 to 10 for improved resilience during service readiness checks.
- Extended the start period from 30s to 60s to allow more time for services to initialize before health checks commence.
* Add step to create test configuration file in robot-tests.yml
- Introduced a new step in the GitHub Actions workflow to copy the test configuration file from tests/configs/deepgram-openai.yml to a new config/config.yml.
- Added logging to confirm the creation of the test config file, improving visibility during the test setup process.
* remove cache step since not required
* coderabbit comments
* Refactor ConfigManager error handling for configuration file loading
- Updated the ConfigManager to raise RuntimeError exceptions when the configuration file is not found or is invalid, improving error visibility and user guidance.
- Removed fallback behavior that previously returned the current directory, ensuring users are explicitly informed about missing or invalid configuration files.
* Refactor _find_repo_root method in ConfigManager
- Updated the _find_repo_root method to locate the repository root using the __file__ location instead of searching for config/config.yml, simplifying the logic and improving reliability.
- Removed the previous error handling that raised a RuntimeError if the configuration file was not found, as the new approach assumes config_manager.py is always at the repo root.
* Enhance speaker recognition service integration and error handling (#245)
* Enhance speaker recognition service integration and error handling
- Updated `docker-compose-test.yml` to enable speaker recognition in the test environment and added a new `speaker-service-test` service for testing purposes.
- Refactored `run-test.sh` to improve the execution of Robot Framework tests from the repository root.
- Enhanced error handling in `speaker_recognition_client.py` to return detailed error messages for connection issues.
- Improved error logging in `speaker_jobs.py` to handle and report errors from the speaker recognition service more effectively.
- Updated `Dockerfile` to copy the full source code after dependencies are cached, ensuring all necessary files are included in the image.
* Remove integration tests workflow and enhance robot tests with HF_TOKEN verification
- Deleted the `integration-tests.yml` workflow file to streamline CI processes.
- Updated `robot-tests.yml` to include verification for the new `HF_TOKEN` secret, ensuring all required secrets are checked before running tests.
* Fix key access in system admin tests to use string indexing for speakers data
* Refactor Robot Framework tests and enhance error handling in memory services
- Removed the creation of the test environment file from the GitHub Actions workflow to streamline setup.
- Updated the Robot Framework tests to utilize a unified test script for improved consistency.
- Enhanced error messages in the MemoryService class to provide more context on connection failures for LLM and vector store providers.
- Added critical checks for API key presence in the OpenAIProvider class to ensure valid credentials are provided before proceeding.
- Adjusted various test setup scripts to use a centralized BACKEND_DIR variable for better maintainability and clarity.
* Refactor test container cleanup in run-robot-tests.sh
- Updated the script to dynamically construct container names from docker-compose services, improving maintainability and reducing hardcoded values.
- Enhanced the cleanup process for stuck test containers by utilizing the COMPOSE_PROJECT_NAME variable.
* Enhance run-robot-tests.sh for improved logging and cleanup
- Set absolute paths for consistent directory references to simplify navigation.
- Capture container logs, status, and resource usage for better debugging.
- Refactor cleanup process to utilize dynamic backend directory references, improving maintainability.
- Ensure proper navigation back to the tests directory after operations.
* Add speaker recognition configuration and update test script defaults
- Introduced speaker recognition settings in config.yml.template, allowing for easy enable/disable and service URL configuration.
- Updated run-robot-tests.sh to use a test-specific configuration file that disables speaker recognition for improved CI performance.
- Modified deepgram-openai.yml to disable speaker recognition during CI tests to enhance execution speed.
* Refactor speaker recognition configuration management
- Updated docker-compose-test.yml to clarify speaker recognition settings, now controlled via config.yml for improved CI performance.
- Enhanced model_registry.py to include a dedicated speaker_recognition field for better configuration handling.
- Modified speaker_recognition_client.py to load configuration from config.yml, allowing for dynamic enabling/disabling of the speaker recognition service based on the configuration.
* Add minimum worker count verification to infrastructure tests
- Introduced a new keyword to verify that the minimum number of workers are registered, enhancing the robustness of health checks.
- Updated the worker count validation test to include a wait mechanism for worker registration, improving test reliability.
- Clarified comments regarding expected worker counts to reflect the distinction between RQ and audio stream workers.
* Update configuration management and enhance model handling
- Added OBSIDIAN_ENABLED configuration to ChronicleSetup for improved feature toggling.
- Introduced speaker_recognition configuration handling in model_registry.py to streamline model loading.
- Refactored imports in deepgram.py to improve clarity and reduce redundancy.
* Refactor configuration management in wizard and ChronicleSetup (#246)
* Refactor configuration management in wizard and ChronicleSetup
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling.
- Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth.
- Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
* Fix string formatting for error message in ChronicleSetup
* added JWT issuers for audience auth for service interop and shared us… (#250)
* added JWT issuers for audience auth for service interop and shared user accounts
* amended default value in line wioth code
* Feat/edit chat system prompt (#247)
* Refactor configuration management in wizard and ChronicleSetup
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling.
- Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth.
- Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
* Fix string formatting for error message in ChronicleSetup
* Enhance chat configuration management and UI integration
- Updated `services.py` to allow service restart with an option to recreate containers, addressing WSL2 bind mount issues.
- Added new chat configuration management functions in `system_controller.py` for loading, saving, and validating chat prompts.
- Introduced `ChatSettings` component in the web UI for admin users to manage chat configurations easily.
- Updated API service methods in `api.ts` to support chat configuration endpoints.
- Integrated chat settings into the system management page for better accessibility.
* Refactor backend shutdown process and enhance chat service configuration logging
- Updated `start.sh` to improve shutdown handling by explicitly killing the backend process if running.
- Modified `chat_service.py` to enhance logging for loading chat system prompts, providing clearer feedback on configuration usage.
- Added a new `chat` field in `model_registry.py` for better chat service configuration management.
- Updated vector store query parameters in `vector_stores.py` for improved clarity and functionality.
- Enhanced the chat component in the web UI to conditionally auto-scroll based on message sending status.
* Return JSONResponse instead of raw result
* Refactor headers creation in system admin tests
* Make config.yml writable for admin updates
* Docs consolidation (#257)
* Enhance setup documentation and convenience scripts
- Updated the interactive setup wizard instructions to recommend using the convenience script `./wizard.sh` for easier configuration.
- Added detailed instructions for uploading and processing existing audio files via the API, including example commands for single and multiple file uploads.
- Introduced a new section on HAVPE relay configuration for ESP32 audio streaming, providing environment variable setup and command examples.
- Clarified the distributed deployment setup, including GPU and backend separation instructions, and added benefits of using Tailscale for networking.
- Removed outdated `getting-started.md` and `SETUP_SCRIPTS.md` files to streamline documentation and avoid redundancy.
* Update setup instructions and enhance service management scripts
- Replaced direct command instructions with convenience scripts (`./wizard.sh` and `./start.sh`) for easier setup and service management.
- Added detailed usage of convenience scripts for checking service status, restarting, and stopping services.
- Clarified the distinction between convenience scripts and direct command usage for improved user guidance.
* Update speaker recognition models and documentation
- Changed the speaker diarization model from `pyannote/speaker-diarization-3.1` to `pyannote/speaker-diarization-community-1` across multiple files for consistency.
- Updated README files to reflect the new model and its usage instructions, ensuring users have the correct links and information for setup.
- Enhanced clarity in configuration settings related to speaker recognition.
* Docs consolidation (#258)
* Enhance setup documentation and convenience scripts
- Updated the interactive setup wizard instructions to recommend using the convenience script `./wizard.sh` for easier configuration.
- Added detailed instructions for uploading and processing existing audio files via the API, including example commands for single and multiple file uploads.
- Introduced a new section on HAVPE relay configuration for ESP32 audio streaming, providing environment variable setup and command examples.
- Clarified the distributed deployment setup, including GPU and backend separation instructions, and added benefits of using Tailscale for networking.
- Removed outdated `getting-started.md` and `SETUP_SCRIPTS.md` files to streamline documentation and avoid redundancy.
* Update setup instructions and enhance service management scripts
- Replaced direct command instructions with convenience scripts (`./wizard.sh` and `./start.sh`) for easier setup and service management.
- Added detailed usage of convenience scripts for checking service status, restarting, and stopping services.
- Clarified the distinction between convenience scripts and direct command usage for improved user guidance.
* Update speaker recognition models and documentation
- Changed the speaker diarization model from `pyannote/speaker-diarization-3.1` to `pyannote/speaker-diarization-community-1` across multiple files for consistency.
- Updated README files to reflect the new model and its usage instructions, ensuring users have the correct links and information for setup.
- Enhanced clarity in configuration settings related to speaker recognition.
* Enhance transcription provider selection and update HTTPS documentation
- Added a new function in `wizard.py` to prompt users for their preferred transcription provider, allowing options for Deepgram, Parakeet ASR, or none.
- Updated the service setup logic to automatically include ASR services if Parakeet is selected.
- Introduced a new documentation file on SSL certificates and HTTPS setup, detailing the importance of HTTPS for secure connections and microphone access.
- Removed outdated HTTPS setup documentation from `backends/advanced/Docs/HTTPS_SETUP.md` to streamline resources.
* Remove HTTPS setup scripts and related configurations
- Deleted `init-https.sh`, `setup-https.sh`, and `nginx.conf.template` as part of the transition to a new HTTPS setup process.
- Updated `README.md` to reflect the new automatic HTTPS configuration via the setup wizard.
- Adjusted `init.py` to remove references to the deleted HTTPS scripts and ensure proper handling of Caddyfile generation for SSL.
- Streamlined documentation to clarify the new approach for HTTPS setup and configuration management.
* Update quickstart.md (#268)
* v0.2 (#279)
* Refactor configuration management in wizard and ChronicleSetup
- Updated wizard.py to read Obsidian/Neo4j configuration from config.yml, enhancing flexibility and error handling.
- Refactored ChronicleSetup to utilize ConfigManager for loading and verifying config.yml, ensuring a single source of truth.
- Improved user feedback for missing configuration files and streamlined the setup process for memory and transcription providers.
* Fix string formatting for error message in ChronicleSetup
* Enhance chat configuration management and UI integration
- Updated `services.py` to allow service restart with an option to recreate containers, addressing WSL2 bind mount issues.
- Added new chat configuration management functions in `system_controller.py` for loading, saving, and validating chat prompts.
- Introduced `ChatSettings` component in the web UI for admin users to manage chat configurations easily.
- Updated API service methods in `api.ts` to support chat configuration endpoints.
- Integrated chat settings into the system management page for better accessibility.
* Refactor backend shutdown process and enhance chat service configuration logging
- Updated `start.sh` to improve shutdown handling by explicitly killing the backend process if running.
- Modified `chat_service.py` to enhance logging for loading chat system prompts, providing clearer feedback on configuration usage.
- Added a new `chat` field in `model_registry.py` for better chat service configuration management.
- Updated vector store query parameters in `vector_stores.py` for improved clarity and functionality.
- Enhanced the chat component in the web UI to conditionally auto-scroll based on message sending status.
* Implement plugin system for enhanced functionality and configuration management
- Introduced a new plugin architecture to allow for extensibility in the Chronicle application.
- Added Home Assistant plugin for controlling devices via natural language commands triggered by wake words.
- Implemented plugin configuration management endpoints in the API for loading, saving, and validating plugin settings.
- Enhanced the web UI with a dedicated Plugins page for managing plugin configurations.
- Updated Docker Compose files to include Tailscale integration for remote service access.
- Refactored existing services to support plugin interactions during conversation and memory processing.
- Improved error handling and logging for plugin initialization and execution processes.
* Enhance configuration management and plugin system integration
- Updated .gitignore to include plugins.yml for security reasons.
- Modified start.sh to allow passing additional arguments during service startup.
- Refactored wizard.py to support new HF_TOKEN configuration prompts and improved handling of wake words in plugin settings.
- Introduced a new setup_hf_token_if_needed function to streamline Hugging Face token management.
- Enhanced the GitHub Actions workflow to create plugins.yml from a template, ensuring proper configuration setup.
- Added detailed comments and documentation in the plugins.yml.template for better user guidance on Home Assistant integration.
* Implement Redis integration for client-user mapping and enhance wake word processing
- Added asynchronous Redis support in ClientManager for tracking client-user relationships.
- Introduced `initialize_redis_for_client_manager` to set up Redis for cross-container mapping.
- Updated `create_client_state` to use asynchronous tracking for client-user relationships.
- Enhanced wake word processing in PluginRouter with normalization and command extraction.
- Refactored DeepgramStreamingConsumer to utilize async Redis lookups for user ID retrieval.
- Set TTL on Redis streams during client state cleanup for better resource management.
* Refactor Deepgram worker management and enhance text normalization
- Disabled the batch Deepgram worker in favor of the streaming worker to prevent race conditions.
- Updated text normalization in wake word processing to replace punctuation with spaces, preserving word boundaries.
- Enhanced regex pattern for wake word matching to allow optional punctuation and whitespace after the last part.
- Improved logging in DeepgramStreamingConsumer for better visibility of message processing and error handling.
* Add original prompt retrieval and restoration in chat configuration test
- Implemented retrieval of the original chat prompt before saving a custom prompt to ensure test isolation.
- Added restoration of the original prompt after the test to prevent interference with subsequent tests.
- Enhanced the test documentation for clarity on the purpose of these changes.
* Refactor test execution and enhance documentation for integration tests
- Simplified test execution commands in CLAUDE.md and quickstart.md for better usability.
- Added instructions for running tests from the project root and clarified the process for executing the complete Robot Framework test suite.
- Introduced a new Docker service for the Deepgram streaming worker in docker-compose-test.yml to improve testing capabilities.
- Updated system_admin_tests.robot to use a defined default prompt for restoration, enhancing test reliability and clarity.
* Enhance test environment cleanup and improve Deepgram worker management
- Updated `run-test.sh` and `run-robot-tests.sh` to improve cleanup processes, including handling permission issues with Docker.
- Introduced a new function `mark_session_complete` in `session_controller.py` to ensure atomic updates for session completion status.
- Refactored WebSocket and conversation job handling to utilize the new session completion function, enhancing reliability.
- Updated `start-workers.sh` to enable the batch Deepgram worker alongside the streaming worker for improved transcription capabilities.
- Enhanced test scripts to verify the status of Deepgram workers and ensure proper cleanup of test containers.
* Refactor worker management and introduce orchestrator for improved process handling
- Replaced the bash-based `start-workers.sh` script with a Python-based worker orchestrator for better process management and health monitoring.
- Updated `docker-compose.yml` to configure the new orchestrator and adjust worker definitions, including the addition of audio persistence and stream workers.
- Enhanced the Dockerfile to remove the old startup script and ensure the orchestrator is executable.
- Introduced new modules for orchestrator configuration, health monitoring, process management, and worker registry to streamline worker lifecycle management.
- Improved environment variable handling for worker configuration and health checks.
* oops
* oops2
* Remove legacy test runner script and update worker orchestration
- Deleted the `run-test.sh` script, which was used for local test execution.
- Updated Docker configurations to replace the `start-workers.sh` script with `worker_orchestrator.py` for improved worker management.
- Enhanced health monitoring and process management in the orchestrator to ensure better reliability and logging.
- Adjusted deployment configurations to reflect the new orchestrator setup.
* Add bulk restart mechanism for RQ worker registration loss
- Introduced a new method `_handle_registration_loss` to manage RQ worker registration loss, replicating the behavior of the previous bash script.
- Implemented a cooldown period to prevent frequent restarts during network issues.
- Added logging for bulk restart actions and their outcomes to enhance monitoring and debugging capabilities.
- Created a `_restart_all_rq_workers` method to facilitate the bulk restart of RQ workers, ensuring they re-register with Redis upon startup.
* Enhance plugin architecture with event-driven system and test integration
- Introduced a new Test Event Plugin to log all plugin events to an SQLite database for integration testing.
- Updated the plugin system to utilize event subscriptions instead of access levels, allowing for more flexible event handling.
- Refactored the PluginRouter to dispatch events based on subscriptions, improving the event-driven architecture.
- Enhanced Docker configurations to support development and testing environments with appropriate dependencies.
- Added comprehensive integration tests to verify the functionality of the event dispatch system and plugin interactions.
- Updated documentation and test configurations to reflect the new event-based plugin structure.
* Enhance Docker configurations and startup script for test mode
- Updated `docker-compose-test.yml` to include a test command for services, enabling a dedicated test mode.
- Modified `start.sh` to support a `--test` flag, allowing the FastAPI backend to run with test-specific configurations.
- Adjusted worker commands to utilize the `--group test` option in test mode for improved orchestration and management.
* Refactor test scripts for improved reliability and clarity
- Updated `run-robot-tests.sh` to enhance the verification of the Deepgram batch worker process, ensuring non-numeric characters are removed from the check.
- Modified `plugin_tests.robot` to use a more explicit method for checking the length of subscriptions and added a skip condition for unavailable audio files.
- Adjusted `plugin_event_tests.robot` to load the test audio file from a variable, improving test data management.
- Refactored `plugin_keywords.robot` to utilize clearer length checks for subscriptions and event parts, enhancing readability and maintainability.
* remove mistral deadcode; notebooks untouched
* Refactor audio streaming endpoints and improve documentation
- Updated WebSocket endpoints to use a unified format with codec parameters (`/ws?codec=pcm` and `/ws?codec=opus`) for audio streaming, replacing the previous `/ws_pcm` and `/ws_omi` endpoints.
- Enhanced documentation to reflect the new endpoint structure and clarify audio processing capabilities.
- Removed deprecated audio cropping functionality and related configurations to streamline the audio processing workflow.
- Updated various components and scripts to align with the new endpoint structure, ensuring consistent usage across the application.
* Enhance testing infrastructure and API routes for plugin events
- Updated `docker-compose-test.yml` to introduce low speech detection thresholds for testing, improving the accuracy of speech detection during tests.
- Added new test-only API routes in `test_routes.py` for clearing and retrieving plugin events, ensuring a clean state between tests.
- Refactored existing test scripts to utilize the new API endpoints for event management, enhancing test reliability and clarity.
- Improved logging and error handling in various components to facilitate debugging during test execution.
- Adjusted environment variable handling in test setup scripts to streamline configuration and improve flexibility.
* Add audio pipeline architecture documentation and improve audio persistence worker configuration
- Introduced a comprehensive documentation file detailing the audio pipeline architecture, covering data flow, processing stages, and key components.
- Enhanced the audio persistence worker setup by implementing multiple concurrent workers to improve audio processing efficiency.
- Adjusted sleep intervals in the audio streaming persistence job for better responsiveness and event loop yielding.
- Updated test script to run the full suite of integration tests from the specified directory, ensuring thorough testing coverage.
* Add test container setup and teardown scripts
- Introduced `setup-test-containers.sh` for streamlined startup of test containers, including health checks and environment variable loading.
- Added `teardown-test-containers.sh` for simplified container shutdown, with options to remove volumes.
- Enhanced user feedback with color-coded messages for better visibility during test setup and teardown processes.
* Update worker count validation and websocket disconnect tests
- Adjusted worker count expectations in the Worker Count Validation Test to reflect an increase from 7 to 9 workers, accounting for additional audio persistence workers.
- Enhanced the WebSocket Disconnect Conversation End Reason Test by adding steps to maintain audio streaming during disconnection, ensuring accurate simulation of network dropout scenarios.
- Improved comments for clarity and added critical notes regarding inactivity timeout handling.
* Refactor audio storage to MongoDB chunks and enhance cleanup settings management
- Replaced the legacy AudioFile model with AudioChunkDocument for storing audio data in MongoDB, optimizing storage and retrieval.
- Introduced CleanupSettings dataclass for managing soft-deletion configurations, including auto-cleanup and retention days.
- Added admin API routes for retrieving and saving cleanup settings, ensuring better control over data retention policies.
- Updated audio processing workflows to utilize MongoDB chunks, removing dependencies on disk-based audio files.
- Enhanced tests to validate the new audio chunk storage and cleanup functionalities, ensuring robust integration with existing systems.
* Refactor audio processing to utilize MongoDB chunks and enhance job handling
- Removed audio file path parameters from various functions, transitioning to audio data retrieval from MongoDB chunks.
- Updated the `start_post_conversation_jobs` function to reflect changes in audio handling, ensuring jobs reconstruct audio from database chunks.
- Enhanced the `transcribe_full_audio_job` and `recognise_speakers_job` to process audio directly from memory, eliminating the need for temporary files.
- Improved error handling and logging for audio data retrieval, ensuring better feedback during processing.
- Added a new utility function for converting PCM data to WAV format in memory, streamlining audio format handling.
* Refactor speaker recognition client to use in-memory audio data
- Updated methods to accept audio data as bytes instead of file paths, enhancing performance by eliminating disk I/O.
- Improved logging to reflect in-memory audio processing, providing better insights during speaker identification and diarization.
- Streamlined audio data handling in the `diarize_identify_match` and `diarize_and_identify` methods, ensuring consistency across the client.
- Removed temporary file handling, simplifying the audio processing workflow and reducing potential file system errors.
* Add mock providers and update testing workflows for API-independent execution
- Introduced `MockLLMProvider` and `MockTranscriptionProvider` to facilitate testing without external API dependencies, allowing for consistent and controlled test environments.
- Created `run-no-api-tests.sh` script to execute tests that do not require API keys, ensuring separation of API-dependent and independent tests.
- Updated Robot Framework test configurations to utilize mock services, enhancing test reliability and reducing external dependencies.
- Modified existing test workflows to include new configurations and ensure proper handling of results for tests excluding API keys.
- Added `mock-services.yml` configuration to disable external API services while maintaining core functionality for testing purposes.
- Enhanced documentation to reflect the new tagging system for tests requiring API keys, improving clarity on test execution requirements.
* Enhance testing documentation and workflows for API key separation
- Updated CLAUDE.md to clarify test execution modes, emphasizing the separation of tests requiring API keys from those that do not.
- Expanded the testing guidelines in TESTING_GUIDELINES.md to detail the organization of tests based on API dependencies, including tagging conventions and execution paths.
- Improved mock-services.yml to include dummy configurations for LLM and embedding services, ensuring tests can run without actual API calls.
- Added comprehensive documentation on GitHub workflows for different test scenarios, enhancing clarity for contributors and maintainers.
* Update test configurations and documentation for API key management
- Modified `plugins.yml.template` to implement event subscriptions for the Home Assistant plugin, enhancing its event-driven capabilities.
- Revised `README.md` to clarify test execution processes, emphasizing the distinction between tests requiring API keys and those that do not.
- Updated `mock-services.yml` to streamline mock configurations, ensuring compatibility with the new testing workflows.
- Added `requires-api-keys` tags to relevant test cases across various test files, improving organization and clarity regarding API dependencies.
- Enhanced documentation for test scripts and configurations, providing clearer guidance for contributors on executing tests based on API key requirements.
* Add optional service profile to Docker Compose test configuration
* Refactor audio processing and job handling for transcription workflows
- Updated `upload_and_process_audio_files` and `start_post_conversation_jobs` to enqueue transcription jobs separately for file uploads, ensuring accurate processing order.
- Enhanced logging to provide clearer insights into job enqueuing and processing stages.
- Removed batch transcription from the post-conversation job chain for streaming audio, utilizing the streaming transcript directly.
- Introduced word-level timestamps in the `Conversation` model to improve transcript detail and accuracy.
- Updated tests to reflect changes in job handling and ensure proper verification of post-conversation processing.
* Remove unnecessary network aliases from speaker service in Docker Compose configuration
* Add network aliases for speaker service in Docker Compose configuration
* Refactor Conversation model to use string for provider field
- Updated the `Conversation` model to replace the `TranscriptProvider` enum with a string type for the `provider` field, allowing for greater flexibility in provider names.
- Adjusted related job functions to accommodate this change, simplifying provider handling in the transcription workflow.
* Enhance configuration and model handling for waveform data
- Updated Docker Compose files to mount the entire config directory, allowing for better management of configuration files.
- Introduced a new `WaveformData` model to store pre-computed waveform visualization data, improving UI performance by enabling waveform display without real-time decoding.
- Enhanced the `app_factory` and `job` models to include the new `WaveformData` model, ensuring proper initialization and data handling.
- Implemented waveform generation logic in a new worker module, allowing for on-demand waveform creation from audio chunks.
- Added API endpoints for retrieving and generating waveform data, improving the overall audio processing capabilities.
- Updated tests to cover new functionality and ensure robustness in waveform data handling.
* Add SDK testing scripts for authentication, conversation retrieval, and audio upload
- Introduced three new test scripts: `sdk_test_auth.py`, `sdk_test_conversations.py`, and `sdk_test_upload.py`.
- Each script tests different functionalities of the SDK, including authentication, conversation retrieval, and audio file uploads.
- The scripts utilize the `ChronicleClient` to perform operations and print results for verification.
- Enhanced testing capabilities for the SDK, ensuring robust validation of core features.
* Enhance audio processing and conversation handling for large files
- Added configuration options for speaker recognition chunking in `.env.template`, allowing for better management of large audio files.
- Updated `get_conversations` function to include an `include_deleted` parameter for filtering conversations based on their deletion status.
- Enhanced `finalize_session` method in `AudioStreamProducer` to send an end marker to Redis, ensuring proper session closure.
- Introduced `reconstruct_audio_segments` function to yield audio segments with overlap for efficient processing of lengthy conversations.
- Implemented merging of overlapping speaker segments to improve accuracy in speaker recognition.
- Added integration tests for WebSocket streaming transcription to validate the end_marker functionality and overall transcription flow.
* archive
* Implement annotation system and enhance audio processing capabilities
- Introduced a new annotation model to support user edits and AI-powered suggestions for memories and transcripts.
- Added annotation routes for CRUD operations, enabling the creation and management of annotations via the API.
- Enhanced the audio processing workflow to support fetching audio segments from the backend, improving speaker recognition accuracy.
- Updated the speaker recognition client to handle conversation-based audio fetching, allowing for better management of large audio files.
- Implemented a cron job for generating AI suggestions on potential errors in transcripts and memories, improving user experience and content accuracy.
- Enhanced the web UI to support inline editing of transcript segments and memory content, providing a more interactive user experience.
- Updated configuration files to support new features and improve overall system flexibility.
* Implement OmegaConf-based configuration management for backend settings
- Introduced a new configuration loader using OmegaConf for unified management of backend settings.
- Updated existing configuration functions to leverage the new loader, enhancing flexibility and maintainability.
- Added support for environment variable interpolation in configuration files.
- Refactored various components to retrieve settings from the new configuration system, improving consistency across the application.
- Updated requirements to include OmegaConf as a dependency.
- Enhanced documentation and comments for clarity on configuration management.
* Refactor .env.template and remove unused diarization configuration
- Updated the .env.template to clarify its purpose for secret values and streamline setup instructions.
- Removed the deprecated diarization_config.json.template file, as it is no longer needed.
- Added new environment variables for Langfuse and Tailscale integration to enhance observability and remote service access.
* Implement legacy environment variable syntax support in configuration loader
- Added custom OmegaConf resolvers to handle legacy ${VAR:-default} syntax for backward compatibility.
- Introduced a preprocessing function to convert legacy syntax in YAML files to OmegaConf-compatible format.
- Updated the load_config function to utilize the new preprocessing for loading defaults and user configurations.
- Enhanced documentation for clarity on the new legacy syntax handling.
* Add plugins configuration path retrieval and refactor usage
- Introduced a new function `get_plugins_yml_path` to centralize the retrieval of the plugins.yml file path.
- Updated `system_controller.py` and `plugin_service.py` to use the new function for improved maintainability and consistency in accessing the plugins configuration.
- Enhanced code clarity by removing hardcoded paths and utilizing the centralized configuration method.
* Unify plugin terminology and fix memory job dependencies
Plugin terminology: subscriptions→events, trigger→condition
Memory jobs: no longer blocked by disabled speaker recognition
* Update Docker Compose configuration and enhance system routes
- Updated Docker Compose files to mount the entire config directory, consolidating configuration management.
- Refactored the `save_diarization_settings` function to improve clarity and maintainability by renaming it to `save_diarization_settings_controller`.
- Enhanced the System component in the web UI to include configuration diagnostics, providing better visibility into system health and issues.
* circular import
* Refactor testing infrastructure and enhance container management
- Updated the testing documentation to reflect a new Makefile-based approach for running tests and managing containers.
- Introduced new scripts for container management, including starting, stopping, restarting, and cleaning containers while preserving logs.
- Added a cleanup script to handle data ownership and permissions correctly.
- Implemented a logging system that saves container logs automatically before cleanup.
- Enhanced the README with detailed instructions for running tests and managing the test environment.
* Add Email Summarizer Plugin and SMTP Email Service
- Introduced the Email Summarizer Plugin that automatically sends email summaries upon conversation completion.
- Implemented SMTP Email Service for sending emails, supporting HTML and plain text formats with TLS/SSL encryption.
- Added configuration options for SMTP settings in the .env.template and plugins.yml.template.
- Created comprehensive documentation for plugin development and usage, including a new plugin generation script.
- Enhanced testing coverage for the Email Summarizer Plugin and SMTP Email Service to ensure reliability and functionality.
* Refactor plugin management and introduce Email Summarizer setup
- Removed the static PLUGINS dictionary and replaced it with a dynamic discovery mechanism for plugins.
- Implemented a new setup process for plugins, allowing for configuration via individual setup scripts.
- Added the Email Summarizer plugin with a dedicated setup script for SMTP configuration.
- Enhanced the main setup flow to support community plugins and their configuration.
- Cleaned up unused functions related to plugin configuration and streamlined the overall plugin setup process.
* Enhance plugin configuration and documentation
- Updated the .env.template to include new configuration options for the Home Assistant and Email Summarizer plugins, including server URLs, tokens, and additional settings.
- Refactored Docker Compose files to correctly mount plugin configuration paths.
- Introduced comprehensive documentation for plugin configuration architecture, detailing the separation of concerns for orchestration, settings, and secrets.
- Added individual configuration files for the Home Assistant and Email Summarizer plugins, ensuring proper management of non-secret settings and environment variable references.
- Improved the plugin loading process to merge configurations from multiple sources, enhancing flexibility and maintainability.
* Refactor plugin setup process to allow interactive user input
- Updated the plugin setup script to run interactively, enabling plugins to prompt for user input during configuration.
- Removed output capturing to facilitate real-time interaction and improved error messaging to include exit codes for better debugging.
* Add shared setup utilities for interactive configuration
- Introduced `setup_utils.py` containing functions for reading environment variables, prompting user input, and masking sensitive values.
- Refactored existing code in `wizard.py` and `init.py` to utilize these shared utilities, improving code reuse and maintainability.
- Updated documentation to include usage examples for the new utilities in plugin setup scripts, enhancing developer experience and clarity.
* Enhance plugin security architecture and configuration management
- Introduced a three-file separation for plugin configuration to improve security:
- `backends/advanced/.env` for secrets (gitignored)
- `config/plugins.yml` for orchestration with environment variable references
- `plugins/{plugin_id}/config.yml` for non-secret defaults
- Updated documentation to emphasize the importance of using `${ENV_VAR}` syntax for sensitive data and provided examples of correct usage.
- Enhanced the Email Summarizer plugin setup process to automatically update `config/plugins.yml` with environment variable references, ensuring secrets are not hardcoded.
- Added new fields to the User model for notification email management and improved error logging in user-related functions.
- Refactored audio chunk utilities to use a consistent method for fetching conversation metadata.
* Refactor backend components for improved functionality and stability
- Added a new parameter `transcript_version_id` to the `open_conversation_job` function to support streaming transcript versioning.
- Enhanced error handling in `check_enrolled_speakers_job` and `recognise_speakers_job` to allow conversations to proceed even when the speaker service is unavailable, improving resilience.
- Updated `send_to_adv.py` to support dynamic WebSocket and HTTP protocols based on environment settings, enhancing configuration flexibility.
- Introduced a background task in `send_to_adv.py` to handle incoming messages from the backend, ensuring connection stability and logging interim results.
* Refactor plugin setup timing to enhance configuration flow
* Refactor save_diarization_settings_controller to improve validation and error handling
- Updated the controller to filter out invalid settings instead of raising an error for each unknown key, allowing for more flexible input.
- Added a check to reject requests with no valid settings provided, enhancing robustness.
- Adjusted logging to reflect the filtered settings being saved.
* Refactor audio processing and conversation management for improved deduplication and tracking
* Refactor audio and email handling for improved functionality and security
- Updated `mask_value` function to handle whitespace more effectively.
- Enhanced `create_plugin` to remove existing directories when using the `--force` option.
- Changed logging level from error to debug for existing admin user checks.
- Improved client ID generation logging for clarity.
- Removed unused fields from conversation creation.
- Added HTML escaping in email templates to prevent XSS attacks.
- Updated audio file download function to include user ID for better tracking.
- Adjusted WebSocket connection settings to respect SSL verification based on environment variables.
* Refactor audio upload functionality to remove unused parameters
- Removed `auto_generate_client` and `folder` parameters from audio upload functions to streamline the API.
- Updated related function calls and documentation to reflect these changes, enhancing clarity and reducing complexity.
* Refactor Email Summarizer plugin configuration for improved clarity and security
- Removed outdated migration instructions from `plugin-configuration.md` to streamline documentation.
- Enhanced `README.md` to clearly outline the three-file separation for plugin configuration, emphasizing the roles of `.env`, `config.yml`, and `plugins.yml`.
- Updated `setup.py` to reflect changes in orchestration settings, ensuring only relevant configurations are included in `config/plugins.yml`.
- Improved security messaging to highlight the importance of not committing secrets to version control.
* Update API key configuration in config.yml.template to use environment variable syntax for improved flexibility and security. This change standardizes the way API keys are referenced across different models and services. (#273)
Co-authored-by: roshan.john <roshanjohn1460@gmail.com>
* Refactor Redis job queue cleanup process for improved success tracking
- Replaced total job count with separate counters for successful and failed jobs during Redis queue cleanup.
- Enhanced logging to provide detailed feedback on the number of jobs cleared and any failures encountered.
- Improved error handling to ensure job counts are accurately reflected even when exceptions occur.
* fix tests
* Update CI workflows to use 'docker compose' for log retrieval and added container status check
- Replaced 'docker logs' commands with 'docker compose -f docker-compose-test.yml logs' for consistency across workflows.
- Added a check for running containers before saving logs to enhance debugging capabilities.
* test fixes
* FIX StreamingTranscriptionConsumer to support cumulative audio timestamp adjustments
- Added `audio_offset_seconds` to track cumulative audio duration for accurate timestamp adjustments across transcription sessions.
- Updated `store_final_result` method to adjust word and segment timestamps based on cumulative audio offset.
- Improved logging to reflect changes in audio offset after storing results.
- Modified Makefile and documentation to clarify test execution options, including new tags for slow and SDK tests, enhancing test organization and execution clarity.
* Enhance test container setup and improve error messages in integration tests
- Set `COMPOSE_PROJECT_NAME` for test containers to ensure consistent naming.
- Consolidated error messages in the `websocket_transcription_e2e_test.robot` file for clarity, improving readability and debugging.
* Improve WebSocket closing logic and enhance integration test teardown
- Added timeout handling for WebSocket closure in `AudioStreamClient` to prevent hanging and ensure clean disconnection.
- Updated integration tests to log the total chunks sent when closing audio streams, improving clarity on resource management during test teardown.
* Refactor job status handling to align with RQ standards
- Updated job status checks across various modules to use "started" and "finished" instead of "processing" and "completed" for consistency with RQ's naming conventions.
- Adjusted related logging and response messages to reflect the new status terminology.
- Simplified Docker Compose project name handling in test scripts to avoid conflicts and improve clarity in test environment setup.
* Update test configurations and improve audio inactivity handling
- Increased `SPEECH_INACTIVITY_THRESHOLD_SECONDS` to 20 seconds in `docker-compose-test.yml` for better audio duration handling during tests.
- Refactored session handling in `session_controller.py` to clarify client ID usage.
- Updated `conversation_utils.py` to track speech activity using audio timestamps, enhancing accuracy in inactivity detection.
- Simplified test scripts by removing unnecessary `COMPOSE_PROJECT_NAME` references, aligning with the new project naming convention.
- Adjusted integration tests to reflect changes in inactivity timeout and ensure proper handling of audio timestamps.
* Refactor audio processing and enhance error handling
- Updated `worker_orchestrator.py` to use `logger.exception` for improved error logging.
- Changed default MongoDB database name from "friend-lite" to "chronicle" in multiple files for consistency.
- Added a new method `close_stream_without_stop` in `audio_stream_client.py` to handle abrupt WebSocket disconnections.
- Enhanced audio validation in `audio_utils.py` to support automatic resampling of audio data if sample rates do not match.
- Improved logging in various modules to provide clearer insights during audio processing and event dispatching.
* Enhance Docker command handling and configuration management
- Updated `run_compose_command` to support separate build commands for services, including profile management for backend and speaker-recognition services.
- Improved error handling and output streaming during Docker command execution.
- Added `ensure_docker_network` function to verify and create the required Docker network before starting services.
- Refactored configuration files to utilize `oc.env` for environment variable management, ensuring better compatibility and flexibility across different environments.
* Enhance configuration loading to support custom config file paths
- Added support for the CONFIG_FILE environment variable to allow specifying custom configuration files for testing.
- Implemented logic to handle both absolute paths and relative filenames for the configuration file, improving flexibility in configuration management.
* Update test scripts to use TEST_CONFIG_FILE for configuration management
- Replaced CONFIG_FILE with TEST_CONFIG_FILE in both run-no-api-tests.sh and run-robot-tests.sh to standardize configuration file usage.
- Updated paths to point to mock and deepgram-openai configuration files inside the container, improving clarity and consistency in test setups.
* Refactor audio upload response handling and improve error reporting
- Updated `upload_and_process_audio_files` to return appropriate HTTP status codes based on upload results: 400 for all failures, 207 for partial successes, and 200 for complete success.
- Enhanced error messages in the audio upload tests to provide clearer feedback on upload failures, including specific error details for better debugging.
- Adjusted test scripts to ensure consistent handling of conversation IDs in job metadata, improving validation checks for job creation.
* Refactor audio processing and job handling to improve transcription management
- Updated `upload_and_process_audio_files` to check for transcription provider availability before enqueueing jobs, enhancing error handling and logging.
- Modified `start_post_conversation_jobs` to conditionally enqueue memory extraction jobs based on configuration, improving flexibility in job management.
- Enhanced event dispatch job dependencies to only include jobs that were actually enqueued, ensuring accurate job tracking.
- Added `is_transcription_available` function to check transcription provider status, improving modularity and clarity in the transcription workflow.
* Enhance integration tests for plugin events and improve error handling
- Updated integration tests to filter plugin events by conversation ID, ensuring accurate event tracking and reducing noise from fixture events.
- Improved error messages in event verification to include conversation ID context, enhancing clarity during test failures.
- Refactored audio upload handling to check for transcription job creation, allowing for more robust conversation polling and error reporting.
- Added new keyword to verify conversation end reasons, improving test coverage for conversation state validation.
* Enhance speaker recognition testing and audio processing
- Added mock speaker recognition client to facilitate testing without resource-intensive dependencies.
- Updated Docker Compose configurations to include mock speaker client for test environments.
- Refactored audio segment reconstruction to ensure precise clipping based on time boundaries.
- Improved error handling in transcription jobs and speaker recognition workflows to enhance robustness.
- Adjusted integration tests to utilize real-time pacing for audio chunk streaming, improving test accuracy.
* Refactor audio chunk retrieval and enhance logging in audio processing
- Introduced logging for audio chunk requests to improve traceability.
- Replaced manual audio chunk processing with a dedicated `reconstruct_audio_segment` function for better clarity and efficiency.
- Improved error handling during audio reconstruction to provide more informative responses in case of failures.
- Cleaned up imports and removed redundant code related to audio chunk calculations.
* Refactor mock speaker recognition client and improve testing structure
- Replaced direct import of mock client with a structured import from the new testing module.
- Introduced a dedicated `mock_speaker_client.py` to provide a mock implementation for speaker recognition, facilitating testing without heavy dependencies.
- Added an `__init__.py` file in the testing directory to organize testing utilities and mocks.
* Enhance conversation model to include word-level timestamps and improve transcript handling
- Added a new `words` field to the `Conversation` model for storing word-level timestamps.
- Updated methods to handle word data during transcript version creation, ensuring compatibility with speaker recognition.
- Refactored conversation job processing to utilize the new word structure, improving data integrity and access.
- Enhanced speaker recognition job to read words from the new standardized location, ensuring backward compatibility with legacy data.
* Implement speaker reprocessing feature and enhance timeout calculation
- Added a new endpoint to reprocess speaker identification for existing transcripts, creating a new version with re-identified speakers.
- Introduced a method…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
✏️ Tip: You can customize this high-level summary in your review settings.