fix: Plugin System Refactor#301
Conversation
…ction - Replaced references to the chronicle Bluetooth library with friend_lite for device management. - Removed the list_devices function and implemented a new prompt_user_to_pick_device function to enhance user interaction when selecting OMI/Neo devices. - Updated the find_and_set_omi_mac function to utilize the new device selection method, improving the overall flow of device connection. - Added a new scan_devices.py script for quick scanning of neo/neosapien devices, enhancing usability. - Updated README.md to reflect new usage instructions and prerequisites for connecting to OMI devices over Bluetooth. - Enhanced start.sh to ensure proper environment variable setup for macOS users.
…end Lite BLE devices - Introduced the friend-lite-sdk, a Python SDK for OMI/Friend Lite BLE devices, enabling audio streaming, button events, and transcription functionalities. - Added LICENSE and NOTICE files to clarify licensing and attribution. - Created pyproject.toml for package management, specifying dependencies and project metadata. - Developed core modules including bluetooth connection handling, button event parsing, audio decoding, and transcription capabilities. - Implemented example usage in README.md to guide users on installation and basic functionality. - Enhanced connect-omi.py to utilize the new SDK for improved device management and event handling. - Updated requirements.txt to reference the new SDK for local development. This commit lays the foundation for further enhancements and integrations with OMI devices.
- Introduced a new `markers` list in `ClientState` to collect button event data during sessions. - Added `add_marker` method to facilitate the addition of markers to the current session. - Implemented `on_button_event` method in the `BasePlugin` class to handle device button events, providing context data for button state and timestamps. - Updated `PluginRouter` to route button events to the appropriate plugin handler. - Enhanced conversation job handling to attach markers from Redis sessions, improving the tracking of button events during conversations.
- Introduced the Email Summarizer plugin that automatically sends email summaries upon conversation completion. - Implemented SMTP email service for sending formatted HTML and plain text emails. - Added configuration options for SMTP settings and email content in `config.yml`. - Created setup script for easy configuration of SMTP credentials and plugin orchestration. - Enhanced documentation with usage instructions and troubleshooting tips for the plugin. - Updated existing plugin architecture to support new event handling for email summaries.
- Added external plugins directory to Docker Compose files for better plugin management. - Updated environment variables for MongoDB and Redis services to ensure consistent behavior. - Introduced new dependencies in `uv.lock` for improved functionality. - Refactored audio processing to support various audio formats and enhance error handling. - Implemented new plugin event types and services for better integration and communication between plugins. - Enhanced conversation and session management to support new closing mechanisms and event logging.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughAdds plugin event system with button events, PluginServices for inter-plugin calls, Redis-based conversation close signaling and orphan reprocessing, expanded audio format support and conversion, Redis event logging, friend-lite BLE SDK for OMI devices, web UI event/device features, and multiple plugin & worker integrations. Changes
Sequence Diagram(s)sequenceDiagram
participant Device as OMI Device
participant WS as WebSocket Handler
participant Client as ClientState
participant Router as Plugin Router
participant Plugin as Plugin
participant Services as PluginServices
participant SessionCtrl as Session Controller
participant Redis as Redis
Device->>WS: button_event (state, timestamp)
WS->>Client: add_marker(marker)
WS->>Router: dispatch_event(PluginEvent.BUTTON_..., context)
Router->>Plugin: on_button_event(context)
alt action == close_conversation
Plugin->>Services: close_conversation(session_id, BUTTON_CLOSE)
Services->>Redis: connect
Services->>SessionCtrl: request_conversation_close(session_id, reason)
SessionCtrl->>Redis: set conversation_close_requested
Redis-->>Services: ack
Services-->>Plugin: result
else action == call_plugin
Plugin->>Services: call_plugin(target_plugin, action, data)
Services->>Router: lookup target_plugin
Services->>Plugin: on_plugin_action(context)
Plugin-->>Services: PluginResult
end
Plugin-->>Router: PluginResult
Router->>Redis: _log_event(event_summary)
sequenceDiagram
participant ClientFile as File Upload
participant AudioCtrl as Audio Controller
participant Converter as FFmpeg Converter
participant Queue as Job Queue
participant Worker as Transcription Worker
ClientFile->>AudioCtrl: upload(file)
AudioCtrl->>AudioCtrl: validate extension (SUPPORTED_AUDIO_EXTENSIONS)
alt non-wav file
AudioCtrl->>Converter: convert_any_to_wav(data, ext)
Converter-->>AudioCtrl: wav_bytes
end
AudioCtrl->>Queue: enqueue transcription job (wav_bytes, metadata)
Worker->>Worker: read WAV header -> sample_rate
Worker->>Transcription Service: submit with sample_rate
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes 🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
AnkushMalaker
changed the base branch from
dev
to
feat/better-reprocess-memory
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
3 similar comments
✅ Actions performedReview triggered.
|
✅ Actions performedReview triggered.
|
✅ Actions performedReview triggered.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 17
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (5)
extras/speaker-recognition/src/simple_speaker_recognition/api/routers/identification.py (1)
232-238:⚠️ Potential issue | 🟠 MajorRace condition: mutating shared
db.similarity_thris not thread-safe.
db(theUnifiedSpeakerDBsingleton) is shared across all concurrent requests. Temporarily overridingsimilarity_thrand restoring it in afinallyblock creates a TOCTOU race — concurrent requests can observe each other's threshold. The same pattern appears at lines 666–672.Consider passing the threshold as a parameter to
db.identify()instead of mutating shared state.Suggested approach
- original_threshold = db.similarity_thr - db.similarity_thr = threshold - try: - found, speaker_info, confidence = await db.identify(emb, user_id=user_id) - confidence = validate_confidence(confidence, "diarize_and_identify") - finally: - db.similarity_thr = original_threshold + found, speaker_info, confidence = await db.identify(emb, user_id=user_id, similarity_threshold=threshold) + confidence = validate_confidence(confidence, "diarize_and_identify")This requires updating
UnifiedSpeakerDB.identify()to accept an optionalsimilarity_thresholdoverride parameter.backends/advanced/scripts/create_plugin.py (1)
397-400:⚠️ Potential issue | 🟡 MinorStale path in "Resources" output.
Line 400 prints
plugins/base.pyas the base plugin class location, but the actual class lives atadvanced_omi_backend.plugins.base(inside the backend package). This will confuse plugin authors.💡 Suggested fix
- print(f" - Base plugin class: plugins/base.py") + print(f" - Base plugin class: advanced_omi_backend.plugins.base")backends/advanced/src/advanced_omi_backend/controllers/audio_controller.py (1)
142-142:⚠️ Potential issue | 🟡 MinorDead code:
version_idassigned twice.
version_idis first generated at line 142 but never used before being overwritten at line 199. Remove the first assignment.🧹 Suggested fix
- version_id = str(uuid.uuid4()) - # Generate title from filenameAlso applies to: 199-199
extras/local-omi-bt/connect-omi.py (1)
212-224:⚠️ Potential issue | 🔴 CriticalProgram hangs after BLE disconnect —
process_audioandbackend_stream_wrappernever terminate.When the device disconnects,
conn.wait_until_disconnected()completes, butprocess_audio()andbackend_stream_wrapper()are blocked on their respectivequeue.get()calls.asyncio.gatherwaits for all three, so it never returns. Thefinallyblock (Line 224) that pushes theNonesentinel is unreachable — creating a deadlock.Consider using
asyncio.gatherwith task cancellation when the disconnect fires, or switching toTaskGroup/ wrapping with a done-callback that cancels the remaining tasks:Proposed fix sketch
async with OmiConnection(mac_address) as conn: await conn.subscribe_audio(handle_ble_data) await conn.subscribe_button(handle_button_event) - await asyncio.gather( - conn.wait_until_disconnected(), - process_audio(), - backend_stream_wrapper(), - ) + audio_task = asyncio.create_task(process_audio()) + backend_task = asyncio.create_task(backend_stream_wrapper()) + try: + await conn.wait_until_disconnected() + finally: + audio_task.cancel() + backend_task.cancel() + await backend_queue.put(None) + try: + await asyncio.gather(audio_task, backend_task, return_exceptions=True) + except asyncio.CancelledError: + passbackends/advanced/webui/src/pages/Conversations.tsx (1)
276-284:⚠️ Potential issue | 🟡 MinorDouble API call on initial mount.
The
[]effect (Line 277) and the[debugMode]effect (Line 283) both fire on initial render, causingloadConversations()to be called twice on page load. Consider removing theloadConversations()call from the mount-only effect since the[debugMode]effect already covers the initial load.Proposed fix
useEffect(() => { - loadConversations() loadEnrolledSpeakers() }, []) // Reload conversations when debug mode toggles (to include/exclude orphans) useEffect(() => { loadConversations() }, [debugMode])
🤖 Fix all issues with AI agents
In `@backends/advanced/Docs/plugin-development-guide.md`:
- Around line 244-249: The example handler on_button_event references
PluginEvent but doesn't show its import; update the snippet to include the
missing import (e.g., add an import for PluginEvent at the top where other
symbols like PluginContext are imported) so readers can run the example; ensure
the import statement explicitly names PluginEvent (and add PluginContext import
if that isn't already present in the surrounding examples).
In `@backends/advanced/src/advanced_omi_backend/controllers/audio_controller.py`:
- Around line 78-80: The code defensively assigns filename = file.filename or
"unknown" but continues to reference file.filename directly later; update the
per-file loop in the AudioController to consistently use the local filename
variable (not file.filename) for all logging, responses, metadata, error
messages, and any string formatting (replace occurrences like in log calls,
returned dicts, and response messages within that loop); keep the ext extraction
using os.path.splitext(filename.lower()) and ensure any other references (e.g.,
in functions/methods handling the upload loop, response building, and logging)
use filename so "unknown" is returned when original file.filename is None.
In
`@backends/advanced/src/advanced_omi_backend/controllers/conversation_controller.py`:
- Line 95: The log call using logger.info currently references user.id which is
inconsistent with other logs and may expose an internal DB id; update the
logger.info call in conversation_controller.py (the line that logs "Conversation
close requested for client {client_id} by user {user.id}") to use user.user_id
instead (i.e., replace user.id with user.user_id) so it matches the rest of the
file and avoids leaking the internal primary key.
- Around line 493-508: Before modifying chunks, capture the conversation's
original deleted state and deleted_at timestamp (e.g., orig_deleted =
conversation.deleted; orig_deleted_at = conversation.deleted_at); then perform
the chunk restore via AudioChunkDocument.find(...).update_many(...) and set
conversation fields, but wrap conversation.save() in a try/except; on exception,
rollback the chunks by calling AudioChunkDocument.find(...).update_many({"$set":
{"deleted": True, "deleted_at": orig_deleted_at}}) (matching the rollback
pattern used in restore_conversation) and re-raise or handle the error so the
conversation and chunks remain consistent.
In
`@backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.py`:
- Around line 1469-1476: In handle_omi_websocket, add explicit handling for
header["type"] == "button-event" similar to the control/streaming branches:
extract button_data = header.get("data", {}), state = button_data.get("state",
"unknown") and await _handle_button_event(client_state, state, user.user_id,
client_id); remove relying on the generic unknown-event debug log to swallow
these messages so OMI button presses trigger the existing _handle_button_event
handler.
In `@backends/advanced/src/advanced_omi_backend/plugins/services.py`:
- Around line 82-87: The dict merge in the PluginContext creation lets data's
"action" override the explicit action (currently {"action": action, **data});
fix this in the PluginContext call inside the method that builds the context
(where PluginContext is constructed with event=PluginEvent.PLUGIN_ACTION,
data={"action": action, **data}, services=self) by ensuring the explicit action
cannot be overwritten—either merge data first then set "action" last (e.g.,
{**data, "action": action}) or remove any "action" key from the incoming data
before merging so the action variable always wins.
- Around line 56-99: call_plugin currently embeds services=self into
PluginContext allowing unbounded recursive cross-plugin calls; add recursion
protection by tracking call depth or a visited-plugin set and
rejecting/depth-limiting further calls. Modify call_plugin to compute and pass a
depth counter or visited set via PluginContext (e.g., add fields like depth:int
or visited:Set[str]) and increment/append before invoking
plugin.on_plugin_action; update the check at the start of call_plugin to return
a PluginResult error when depth exceeds a safe limit or the target plugin id is
already in visited. Ensure any subsequent cross-calls use the same propagated
context (or wrap self with a scoped PluginServices that enforces/propagates
depth/visited) so recursion is prevented across nested plugin.on_plugin_action
calls.
In `@backends/advanced/src/advanced_omi_backend/routers/modules/queue_routes.py`:
- Around line 1061-1072: get_dashboard_data currently calls fetch_events()
unconditionally and includes its results in the dashboard response, creating an
auth inconsistency with the admin-only /queue/events route; modify
get_dashboard_data so it only calls or includes fetch_events() (the nested async
function fetch_events) when the requesting user passes the same admin check used
by the /queue/events endpoint (e.g., check is_superuser on the request/user
object), otherwise set events to an empty list or omit them; ensure you
reference the same authorization logic (is_superuser check) used by the
/queue/events handler so both endpoints return events only for admins.
In `@backends/advanced/src/advanced_omi_backend/services/plugin_service.py`:
- Around line 489-492: The code currently does sys.path.insert(0,
plugins_dir_str) in plugin_service.py which gives plugin packages top import
priority and can shadow stdlib modules; change this to
sys.path.append(plugins_dir_str) so plugins_dir_str (and the plugins directory)
is added at the end of sys.path, keeping standard library/import precedence.
Update the logic around plugins_dir_str (the existing check "if plugins_dir_str
not in sys.path") to still guard duplicates and use sys.path.append instead of
sys.path.insert to avoid shadowing.
In `@backends/advanced/src/advanced_omi_backend/workers/conversation_jobs.py`:
- Around line 428-436: The code incorrectly marks a close-request as an
inactivity timeout by setting timeout_triggered = True in the block that handles
conversation_close_requested; update conversation_jobs.py so that when
redis_client.hget(..., "conversation_close_requested") returns a value you do
NOT set timeout_triggered, and instead either leave timeout_triggered unchanged
or set a dedicated flag like close_requested = True; ensure finalize_received
remains True and the conversation_close_requested key is deleted and
logger.info(...) still runs so downstream metadata reflects a close request
rather than a timeout (refer to the variables/functions: redis_client.hget,
session_key, "conversation_close_requested", timeout_triggered,
finalize_received, and logger.info).
- Around line 308-319: After successfully reading and attaching markers to the
Conversation (see session_key, redis_client.hget, conversation.markers and
conversation.save), delete the markers field from Redis to prevent reuse by
subsequent conversations: call the Redis hash delete (e.g., await
redis_client.hdel(session_key, "markers")) only after json.loads and await
conversation.save() succeed, and log success or warn on deletion failure; do not
delete on parse/save errors so debugging information remains available.
In `@extras/friend-lite-sdk/friend_lite/bluetooth.py`:
- Around line 28-39: The connect method currently leaves self._client set to a
BleakClient instance if await self._client.connect() raises, causing later
no-ops and invalid subscribe() calls; wrap the connect call in a try/except (or
try/finally) inside connect() to handle failures: if BleakClient.connect()
raises, attempt to cleanly disconnect/close the partial client if possible
(check client.is_connected() and await disconnect()), then set self._client back
to None and re-raise the exception so callers see the failure; keep the
_on_disconnect callback as-is but ensure cleanup logic targets the same
self._client instance created before calling connect().
In `@extras/friend-lite-sdk/friend_lite/decoder.py`:
- Around line 8-16: The length check in decode_packet is applied unconditionally
causing packets to be dropped even when strip_header is False; update
decode_packet so the guard depends on strip_header (e.g. if strip_header:
require len(data) > 3 else require len(data) > 0 or non-empty) and only return
b'' when the appropriate minimum length isn't met, then proceed to set
clean_data = bytes(data[3:]) when strip_header is True or clean_data = data when
False; ensure you reference the existing decode_packet function and the
strip_header/local variable names when making the change.
In `@extras/friend-lite-sdk/friend_lite/discover_characteristics.py`:
- Line 19: The module calls asyncio.run(main()) at import time which triggers a
BLE connection; guard that call so it only runs when executed as a script by
wrapping the invocation in if __name__ == "__main__": asyncio.run(main()).
Locate the top-level asyncio.run(main()) call in discover_characteristics.py and
move it under that main guard (keeping the existing async def main() unchanged)
so imports no longer start the BLE connection.
In `@extras/friend-lite-sdk/friend_lite/transcribe.py`:
- Around line 43-50: send_audio currently pulls from audio_queue and blindly
forwards chunks to ws.send, but transcribe_wyoming treats a None sentinel as
termination; update send_audio to check if chunk is None, break/return (closing
or stopping the send loop) instead of calling ws.send(None), and ensure any
necessary cleanup (e.g., closing ws or signaling EOF) happens consistently with
transcribe_wyoming's termination behavior; locate the async def send_audio, the
audio_queue usage and ws.send calls to implement this sentinel handling.
- Around line 90-100: The connection creation and await client.connect() happen
before the inner try, so connection failures escape the loop and the client
isn't cleaned up; move the AsyncClient.from_uri(wyoming_url) and await
client.connect() inside the try block that currently begins at the inner try
(the block surrounding the rest of the per-connection logic), and ensure the
finally section always checks and disconnects client (referencing variables
AsyncClient.from_uri, client.connect, the while True loop, and the
try/except/finally) so connection errors are caught, retried by the while loop,
and any partially created client is properly disconnected.
In `@extras/local-omi-bt/send_to_adv.py`:
- Around line 39-55: The send_button_event function currently calls await
_active_websocket.send(...) without guarding for a closed/failed socket; wrap
the send in a try/except that catches ConnectionClosed (from the websockets lib)
and a general Exception, log the error with context (include button_state and
exception), and clear/reset the shared _active_websocket reference (and close it
if appropriate) so subsequent calls won't reuse a broken socket; keep the rest
of the function flow the same and ensure the error path does not re-raise.
🧹 Nitpick comments (30)
extras/friend-lite-sdk/friend_lite/decoder.py (1)
22-23: Uselogginginstead ofThis module is part of a distributable SDK (
friend-lite-sdk). Usingloggingmodule.Proposed fix
Add at the top of the file:
import logging logger = logging.getLogger(__name__)Then replace the print:
except Exception as e: - print("Opus decode error:", e) + logger.warning("Opus decode error: %s", e) return b''backends/advanced/webui/src/contexts/RecordingContext.tsx (1)
355-356: Replace non-null assertion with a runtime guard.If
audioContextRef.currentis evernullhere (e.g. after a future refactor changes call order), the non-null assertion will produce a confusing runtime error downstream rather than a clear message.Suggested fix
- // Reuse the AudioContext created in startRecording - const audioContext = audioContextRef.current! + // Reuse the AudioContext created in startRecording + const audioContext = audioContextRef.current + if (!audioContext) { + throw new Error('AudioContext not initialized – startRecording must be called first') + }extras/friend-lite-sdk/friend_lite/transcribe.py (5)
3-14: Remove leftover development comments.Lines 3, 9, and 14 have stale inline comments (
# Add logging import,# Correct import,# Added import) that appear to be development notes rather than meaningful documentation.Proposed cleanup
-import logging # Add logging import -import time # Import time for windowing +import logging +import time from asyncio import Queue from contextlib import suppress import websockets -from websockets.exceptions import ( # Correct import +from websockets.exceptions import ( ConnectionClosedError, ConnectionClosedOK, WebSocketException, ) -from wyoming.asr import Transcribe # Added import +from wyoming.asr import Transcribe
38-83: Useloggerinstead of
transcribe_wyominguses the module-levelloggerthroughout, buttranscribeuses bareprint()calls exclusively. This makes log filtering, level control, and structured logging impossible for the Deepgram path.
170-173: Remove leftover# DEBUGannotations.These inline
# DEBUGcomments on the logging statements are development artifacts. The log level (logger.debug) already conveys the intent.
176-182: Transcripts are logged but never surfaced to the caller.Both
transcribe(line 65) andtranscribe_wyoming(line 179) only print/log the received transcript. Neither function returns, yields, or enqueues the result, so callers have no programmatic way to consume transcriptions. If this is intentional for demo purposes, a brief docstring note would help; otherwise, consider accepting an output callback or queue.
234-236: Remove commented-out code.The commented-out
logging.basicConfigon line 235 is dead code. Configuration responsibility belongs to the application entry point, not the library module.extras/friend-lite-sdk/README.md (1)
22-31: Usage example imports unused symbols.
ButtonStateandparse_button_eventare imported but not used in the example snippet. Either trim the import to justOmiConnection, or add a brief button-event example that uses them.extras/friend-lite-sdk/pyproject.toml (1)
20-20:wyomingoptional dependency has no version pin.Unlike the other dependencies,
wyominghas no minimum version constraint, which could lead to compatibility issues if breaking changes are introduced upstream. Consider adding a minimum version.Proposed fix
-wyoming = ["wyoming"] +wyoming = ["wyoming>=1.5.0"](Adjust the version to the minimum you've tested against.)
backends/advanced/src/advanced_omi_backend/models/conversation.py (1)
126-130: Consider a typed marker model instead ofDict[str, Any].Using an untyped
List[Dict[str, Any]]formarkersloses schema validation and makes it harder to enforce consistency across producers (button events, bookmarks, etc.). A lightweight PydanticBaseModelsubclass (e.g.,Markerwith fields liketype,timestamp,data) would provide validation at write time and better discoverability.This is optional — the current approach is flexible and fine for an early iteration.
extras/friend-lite-sdk/friend_lite/discover_characteristics.py (1)
4-4: Hardcoded placeholder MAC address should be parameterized.The MAC address is hardcoded with a "Replace with your actual MAC" comment. Consider accepting it via a CLI argument (
sys.argv[1]) or environment variable to avoid developers accidentally committing their device addresses.plugins/email_summarizer/plugin.py (1)
209-209: User email address logged at INFO level — potential PII compliance concern.
user_emailis logged verbatim in the success message. Consider masking the email (e.g.,u***@gmail.com) in logs to reduce PII exposure, especially in production environments subject to GDPR/CCPA.backends/advanced/src/advanced_omi_backend/controllers/system_controller.py (1)
1105-1105: Importing private function_get_plugins_diracross module boundaries.
_get_plugins_diris a private function (underscore-prefixed) inplugin_service.py, yet it's imported in the controller layer to access the plugins directory path for updating plugin configuration. Consider either renaming it toget_plugins_dir(dropping the underscore) or re-exporting it through the module's public API to signal it's an intentional cross-module contract.extras/friend-lite-sdk/friend_lite/button.py (1)
16-24: Out-of-rangestatevalue will raise an opaqueValueErrorfromIntEnum.If the BLE device sends a state value outside the 0–5 range,
ButtonState(state)raisesValueError: X is not a valid ButtonState. Consider catching this and raising a more descriptive error consistent with the guard on line 22, so callers get a uniform error shape.💡 Suggested improvement
- state, _unused = struct.unpack("<II", data[:8]) - return ButtonState(state) + state, _unused = struct.unpack("<II", data[:8]) + try: + return ButtonState(state) + except ValueError: + raise ValueError(f"Unknown button state: {state}")backends/advanced/webui/src/pages/Upload.tsx (1)
86-90:videoWarningis never cleared when video files are removed individually.When a user removes a video file via the ✕ button (
removeFile),videoWarningstaystrueeven if no video files remain in the list. It's only reset insideclearCompletedwhen the filtered list is empty. Consider also clearing the warning inremoveFile(or deriving it from the file list instead of storing it as independent state).💡 Option: derive from file list instead of separate state
Replace the
videoWarningstate with a derived value:- const [videoWarning, setVideoWarning] = useState(false) + const videoWarning = files.some((f) => { + const ext = '.' + f.file.name.split('.').pop()?.toLowerCase() + return f.file.type.startsWith('video/') || VIDEO_EXTENSIONS.includes(ext) + })Then remove
setVideoWarning(true)fromhandleFileSelectand the reset inclearCompleted.backends/advanced/src/advanced_omi_backend/utils/audio_utils.py (1)
145-152: Consider adding a timeout to the FFmpeg subprocess.
process.communicate()will block indefinitely if FFmpeg hangs (e.g., on a malformed or very large input). A timeout prevents a single bad upload from tying up a worker. The existingresample_audio_with_ffmpeghas the same gap, but sinceconvert_any_to_wavhandles untrusted user uploads of arbitrary formats, it's more exposed.💡 Suggested timeout
- stdout, stderr = await process.communicate(input=file_data) + try: + stdout, stderr = await asyncio.wait_for( + process.communicate(input=file_data), + timeout=300, # 5 minutes + ) + except asyncio.TimeoutError: + process.kill() + raise AudioValidationError(f"FFmpeg conversion timed out for {ext} file")backends/advanced/src/advanced_omi_backend/plugins/base.py (1)
21-21: Consider a type-safe annotation forservices.
Optional[Any]discards all type information. Since this is the public plugin API, a narrower type (even aProtocolor aTYPE_CHECKING-guarded forward reference) would help plugin authors discover the available methods.💡 Example with TYPE_CHECKING guard
+from typing import TYPE_CHECKING +if TYPE_CHECKING: + from advanced_omi_backend.plugins.services import PluginServices `@dataclass` class PluginContext: ... - services: Optional[Any] = None + services: Optional["PluginServices"] = Noneextras/friend-lite-sdk/friend_lite/bluetooth.py (2)
28-46: Reconnection not supported:_disconnectedevent is never reset.If an
OmiConnectionis reused (disconnect → connect again),_disconnectedremains set from the prior disconnect, sowait_until_disconnected()would return immediately. Consider clearing the event at the start ofconnect():async def connect(self) -> None: if self._client is not None: return + self._disconnected.clear()This may not be an issue if the class is only used as a single-use context manager, but it's a latent trap for any future reuse.
Also applies to: 59-63
9-12:print_devices()is a blocking call — consider documenting or providing an async variant.
asyncio.run()will fail if called from within an already-running event loop (e.g., Jupyter, or from an async context). This is fine for CLI usage but worth noting in a docstring.extras/local-omi-bt/requirements.txt (1)
5-5: Relative file reference is CWD-sensitive.
file:../friend-lite-sdkresolves relative to the working directory atpip installtime, not relative torequirements.txt. This works when running fromextras/local-omi-bt/but will break from a different CWD (e.g., repo root in CI). Consider documenting the expected install command in the README, or using an editable install in a setup script.extras/local-omi-bt/connect-omi.py (1)
66-78: Fire-and-forget task forsend_button_event— exceptions will be silently lost.
loop.create_task(send_button_event(state.name))at Line 76 creates a task whose exceptions are never awaited. Any failure insend_button_eventproduces only a "Task exception was never retrieved" warning. For a local tool this is low-impact, but consider adding an error callback or wrapping with a helper:Optional improvement
def handle_button_event(sender: Any, data: bytes) -> None: try: state = parse_button_event(data) except Exception as e: logger.error("Button event parse error: %s", e) return if state != ButtonState.IDLE: logger.info("Button event: %s", state.name) try: loop = asyncio.get_running_loop() - loop.create_task(send_button_event(state.name)) + task = loop.create_task(send_button_event(state.name)) + task.add_done_callback( + lambda t: t.exception() and logger.error("send_button_event failed: %s", t.exception()) + ) except RuntimeError: logger.debug("No running event loop, cannot send button event")backends/advanced/src/advanced_omi_backend/services/transcription/__init__.py (1)
220-233: Unreachable dead code afterraiseinexceptblock.Lines 227–232 follow
raise RuntimeError(...)on line 222 inside theexcept httpx.HTTPStatusErrorhandler, making the debug logging block unreachable. This is pre-existing but worth cleaning up.🧹 Proposed fix
except httpx.HTTPStatusError as e: status = e.response.status_code raise RuntimeError( f"Transcription service '{self._name}' at {url} returned HTTP {status}. " f"{'Check your API key.' if status in (401, 403) else ''}" ) from e - - # DEBUG: Log Deepgram response structure - if "results" in data and "channels" in data.get("results", {}): - channels = data["results"]["channels"] - if channels and "alternatives" in channels[0]: - alt = channels[0]["alternatives"][0] - logger.debug(f"DEBUG Registry: Deepgram alternative keys: {list(alt.keys())}")backends/advanced/webui/src/services/api.ts (1)
342-345: Falsy-check oneventTypewill silently drop empty strings.
eventType && { event_type: eventType }evaluates tofalsefor"". This is likely fine here (empty string shouldn't be sent as a filter), but worth noting the behavior differs from theincludeUnprocessedpattern on Line 113 which uses!== undefined.plugins/test_button_actions/plugin.py (1)
67-72: UnrecognizedButtonActionTypevalues silently returnNone.If a new action type is added to
ButtonActionTypebut not handled here, the method returnsNonewithout any logging. Consider adding a fallback log for forward compatibility.Suggested improvement
elif action_type == ButtonActionType.CALL_PLUGIN: return await self._handle_call_plugin(context, action_config) + logger.warning(f"Unhandled action type: {action_type}") return Nonebackends/advanced/src/advanced_omi_backend/plugins/router.py (1)
98-110: Sync Redis client used in async dispatch context.The
_event_redisclient is synchronous (redis.from_url), and_log_event(Line 310–313) performs a blocking pipeline execute. Sincedispatch_eventisasync, this sync call briefly blocks the event loop. At the current scale (200-entry capped list), the impact is negligible, but consider usingredis.asyncioif event dispatch frequency increases.Also,
_event_redisis never closed duringcleanup_all. Consider closing it there.Suggested cleanup addition
async def cleanup_all(self): """Clean up all registered plugins""" for plugin_id, plugin in self.plugins.items(): try: await plugin.cleanup() logger.info(f"Cleaned up plugin '{plugin_id}'") except Exception as e: logger.error(f"Error cleaning up plugin '{plugin_id}': {e}") + if self._event_redis: + try: + self._event_redis.close() + except Exception: + passbackends/advanced/src/advanced_omi_backend/services/plugin_service.py (1)
28-41: Fragileparents[5]index for repo root resolution.The
parents[5]computation relies on the exact directory depth of this file. If the file is ever moved, this will silently resolve to the wrong directory. The Docker and env-var paths take priority in production, so this primarily affects local development.Consider a more robust approach for local dev
# Local dev: plugin_service.py is at <repo>/backends/advanced/src/advanced_omi_backend/services/ - repo_root = Path(__file__).resolve().parents[5] - return repo_root / "plugins" + # Walk up to find the repo root (look for a known marker file) + current = Path(__file__).resolve().parent + for _ in range(10): + if (current / "plugins").is_dir(): + return current / "plugins" + current = current.parent + # Fallback + return Path(__file__).resolve().parents[5] / "plugins"backends/advanced/src/advanced_omi_backend/plugins/services.py (1)
44-54: New Redis connection created perclose_conversationcall.Each invocation creates and tears down a Redis connection. If button-triggered conversation closes become frequent, consider reusing a connection pool or a persistent async Redis client.
Sketch: lazy-initialized shared connection
+ async def _get_redis(self): + if not hasattr(self, '_redis') or self._redis is None: + import redis.asyncio as aioredis + self._redis = aioredis.from_url(self._redis_url) + return self._redis + async def close_conversation(self, session_id: str, reason=...) -> bool: - import redis.asyncio as aioredis from advanced_omi_backend.controllers.session_controller import ( request_conversation_close, ) - r = aioredis.from_url(self._redis_url) - try: - return await request_conversation_close(r, session_id, reason=reason.value) - finally: - await r.aclose() + r = await self._get_redis() + return await request_conversation_close(r, session_id, reason=reason.value)backends/advanced/src/advanced_omi_backend/plugins/events.py (1)
1-57: Well-structured single source of truth for event constants.Clean design using
(str, Enum)for dual-use as both type-safe identifiers and serializable strings.One minor observation:
ButtonState.LONG_PRESSis defined (Line 35) but intentionally omitted fromBUTTON_STATE_TO_EVENT(Lines 39-42). If this is future work, a brief comment at the mapping would help readers understand it's deliberately unmapped rather than accidentally forgotten.Suggested clarifying comment
# Maps device button states to plugin events +# Note: LONG_PRESS is intentionally unmapped — no plugin event defined yet. BUTTON_STATE_TO_EVENT: Dict[ButtonState, PluginEvent] = { ButtonState.SINGLE_TAP: PluginEvent.BUTTON_SINGLE_PRESS, ButtonState.DOUBLE_TAP: PluginEvent.BUTTON_DOUBLE_PRESS, }backends/advanced/src/advanced_omi_backend/controllers/conversation_controller.py (2)
80-87: Redis client created per request — prefer a shared connection pool.A new
aioredisconnection is instantiated and torn down on every call. Under load this adds latency and may exhaust file descriptors. Consider injecting a shared Redis client (or connection pool) that is initialized at app startup, consistent with how other services in this codebase likely manage Redis.Sketch: inject a shared client
- # Signal the conversation job to close and trigger post-processing - redis_url = os.getenv("REDIS_URL", "redis://localhost:6379/0") - r = aioredis.from_url(redis_url) - try: - success = await request_conversation_close( - r, session_id, reason=ConversationCloseReason.USER_REQUESTED.value - ) - finally: - await r.aclose() + # Signal the conversation job to close and trigger post-processing + from advanced_omi_backend.redis_pool import get_redis # shared pool + r = get_redis() + success = await request_conversation_close( + r, session_id, reason=ConversationCloseReason.USER_REQUESTED.value + )
178-212: Consider consolidating orphan queries into a single$orquery.When
include_unprocessedis true, three separate DB round-trips are made. These could be combined into one query using$orto reduce latency, especially if the database is remote. Not critical given this is a list endpoint, but worth considering.Sketch
- orphan_convs_1 = await Conversation.find(orphan_query_1).sort(-Conversation.created_at).to_list() - orphan_convs_2 = await Conversation.find(orphan_query_2).sort(-Conversation.created_at).to_list() + orphan_convs = await Conversation.find( + {"$or": [orphan_query_1, orphan_query_2]} + ).sort(-Conversation.created_at).to_list()
| ```python | ||
| async def on_button_event(self, context: PluginContext): | ||
| if context.event == PluginEvent.BUTTON_SINGLE_PRESS: | ||
| session_id = context.data.get('session_id') | ||
| await context.services.close_conversation(session_id) | ||
| ``` |
There was a problem hiding this comment.
Button‑event example is missing a PluginEvent import.
The snippet compares against PluginEvent but doesn’t show the import, which can confuse plugin authors.
📝 Suggested doc tweak
+from advanced_omi_backend.plugins.events import PluginEvent
async def on_button_event(self, context: PluginContext):
if context.event == PluginEvent.BUTTON_SINGLE_PRESS:
session_id = context.data.get('session_id')
await context.services.close_conversation(session_id)🤖 Prompt for AI Agents
In `@backends/advanced/Docs/plugin-development-guide.md` around lines 244 - 249,
The example handler on_button_event references PluginEvent but doesn't show its
import; update the snippet to include the missing import (e.g., add an import
for PluginEvent at the top where other symbols like PluginContext are imported)
so readers can run the example; ensure the import statement explicitly names
PluginEvent (and add PluginContext import if that isn't already present in the
surrounding examples).
| # Validate file type | ||
| filename = file.filename or "unknown" | ||
| _, ext = os.path.splitext(filename.lower()) |
There was a problem hiding this comment.
Inconsistent use of filename vs file.filename leads to potential None in responses.
Line 79 defensively assigns filename = file.filename or "unknown", but several downstream references still use file.filename directly (e.g., lines 93, 131, 138, 145, 177, 233, 261, 269). If file.filename is None, these will log or return None instead of "unknown".
💡 Use the `filename` variable consistently
Replace all file.filename references within the per-file loop body with the already-defined filename local variable. For example:
except AudioValidationError as e:
processed_files.append({
- "filename": file.filename,
+ "filename": filename,
"status": "error",
"error": str(e),
})Apply the same pattern to lines 93, 138, 145, 177, 190, 233, 255, 261, 263, 269, 271.
Also applies to: 130-134, 233-233
🤖 Prompt for AI Agents
In `@backends/advanced/src/advanced_omi_backend/controllers/audio_controller.py`
around lines 78 - 80, The code defensively assigns filename = file.filename or
"unknown" but continues to reference file.filename directly later; update the
per-file loop in the AudioController to consistently use the local filename
variable (not file.filename) for all logging, responses, metadata, error
messages, and any string formatting (replace occurrences like in log calls,
returned dicts, and response messages within that loop); keep the ext extraction
using os.path.splitext(filename.lower()) and ensure any other references (e.g.,
in functions/methods handling the upload loop, response building, and logging)
use filename so "unknown" is returned when original file.filename is None.
| status_code=404, | ||
| ) | ||
|
|
||
| logger.info(f"Conversation close requested for client {client_id} by user {user.id}") |
There was a problem hiding this comment.
Inconsistent user identifier in log: user.id vs user.user_id.
Every other log line in this file uses user.user_id. Using user.id here may log an internal database primary key instead of the logical user identifier, creating an inconsistency and a potential minor PII concern if id is sensitive.
Fix
- logger.info(f"Conversation close requested for client {client_id} by user {user.id}")
+ logger.info(f"Conversation close requested for client {client_id} by user {user.user_id}")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| logger.info(f"Conversation close requested for client {client_id} by user {user.id}") | |
| logger.info(f"Conversation close requested for client {client_id} by user {user.user_id}") |
🤖 Prompt for AI Agents
In
`@backends/advanced/src/advanced_omi_backend/controllers/conversation_controller.py`
at line 95, The log call using logger.info currently references user.id which is
inconsistent with other logs and may expose an internal DB id; update the
logger.info call in conversation_controller.py (the line that logs "Conversation
close requested for client {client_id} by user {user.id}") to use user.user_id
instead (i.e., replace user.id with user.user_id) so it matches the rest of the
file and avoids leaking the internal primary key.
| if conversation.deleted: | ||
| await AudioChunkDocument.find( | ||
| AudioChunkDocument.conversation_id == conversation_id, | ||
| AudioChunkDocument.deleted == True, | ||
| ).update_many({"$set": {"deleted": False, "deleted_at": None}}) | ||
|
|
||
| conversation.deleted = False | ||
| conversation.deletion_reason = None | ||
| conversation.deleted_at = None | ||
|
|
||
| # Set processing status and update title | ||
| conversation.processing_status = "reprocessing" | ||
| conversation.title = "Reprocessing..." | ||
| conversation.summary = None | ||
| conversation.detailed_summary = None | ||
| await conversation.save() |
There was a problem hiding this comment.
No rollback for chunk restore if conversation.save() fails.
Lines 494-497 restore soft-deleted audio chunks, and Line 508 saves conversation metadata changes. If save() fails, chunks are already restored but the conversation remains in its previous state (deleted, old processing_status). Compare with restore_conversation (Line 435-447) which has explicit rollback logic for this exact scenario.
Suggested rollback pattern (matching existing `restore_conversation`)
conversation.processing_status = "reprocessing"
conversation.title = "Reprocessing..."
conversation.summary = None
conversation.detailed_summary = None
- await conversation.save()
+ try:
+ await conversation.save()
+ except Exception:
+ logger.error(
+ f"Failed to save orphan conversation {conversation_id}, "
+ "rolling back chunk restores"
+ )
+ if conversation.deleted is False:
+ # We restored chunks above — re-delete them
+ await AudioChunkDocument.find(
+ AudioChunkDocument.conversation_id == conversation_id,
+ AudioChunkDocument.deleted == False,
+ ).update_many({"$set": {"deleted": True, "deleted_at": None}})
+ raise🤖 Prompt for AI Agents
In
`@backends/advanced/src/advanced_omi_backend/controllers/conversation_controller.py`
around lines 493 - 508, Before modifying chunks, capture the conversation's
original deleted state and deleted_at timestamp (e.g., orig_deleted =
conversation.deleted; orig_deleted_at = conversation.deleted_at); then perform
the chunk restore via AudioChunkDocument.find(...).update_many(...) and set
conversation fields, but wrap conversation.save() in a try/except; on exception,
rollback the chunks by calling AudioChunkDocument.find(...).update_many({"$set":
{"deleted": True, "deleted_at": orig_deleted_at}}) (matching the rollback
pattern used in restore_conversation) and re-raise or handle the error so the
conversation and chunks remain consistent.
|
|
||
| elif header["type"] == "button-event": | ||
| button_data = header.get("data", {}) | ||
| button_state = button_data.get("state", "unknown") | ||
| await _handle_button_event( | ||
| client_state, button_state, user.user_id, client_id | ||
| ) | ||
| continue |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# First, find the file and check its size
wc -l backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.pyRepository: SimpleOpenSoftware/chronicle
Length of output: 155
🏁 Script executed:
#!/bin/bash
# Read the OMI websocket handler section (lines 1255-1393)
sed -n '1255,1393p' backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.py | cat -nRepository: SimpleOpenSoftware/chronicle
Length of output: 6501
🏁 Script executed:
#!/bin/bash
# Check around line 1360 to see how unknown event types are handled
sed -n '1350,1370p' backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.py | cat -nRepository: SimpleOpenSoftware/chronicle
Length of output: 957
🏁 Script executed:
#!/bin/bash
# Read lines 1469-1476 to see the button event handling in control/streaming modes
sed -n '1469,1476p' backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.py | cat -nRepository: SimpleOpenSoftware/chronicle
Length of output: 514
🏁 Script executed:
#!/bin/bash
# Read lines 1558-1564 to see the second location where this applies
sed -n '1558,1564p' backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.py | cat -nRepository: SimpleOpenSoftware/chronicle
Length of output: 610
🏁 Script executed:
#!/bin/bash
# Search for all button-event references in the codebase
rg "button-event" --type=py -B 2 -A 2Repository: SimpleOpenSoftware/chronicle
Length of output: 1811
🏁 Script executed:
#!/bin/bash
# Search for Wyoming protocol references and documentation
rg "wyoming" --type=py -i -B 1 -A 1 | head -50Repository: SimpleOpenSoftware/chronicle
Length of output: 2997
🏁 Script executed:
#!/bin/bash
# Search for OMI device specifications or button event documentation
rg "OMI.*button|button.*OMI" --type=py -i -B 2 -A 2Repository: SimpleOpenSoftware/chronicle
Length of output: 4544
Add button-event handling to OMI WebSocket handler.
Button events are correctly handled in control and streaming modes (lines 1469–1476, 1558–1564), but the OMI WebSocket handler (handle_omi_websocket, lines 1255–1393) does not process button-event messages. The else clause at line 1360 logs unknown event types at debug level only, silently discarding them. Since OMI devices can send button events via Wyoming protocol (confirmed in extras/local-omi-bt/send_to_adv.py), button presses from OMI devices will be dropped and never trigger the button action handlers.
🤖 Prompt for AI Agents
In
`@backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.py`
around lines 1469 - 1476, In handle_omi_websocket, add explicit handling for
header["type"] == "button-event" similar to the control/streaming branches:
extract button_data = header.get("data", {}), state = button_data.get("state",
"unknown") and await _handle_button_event(client_state, state, user.user_id,
client_id); remove relying on the generic unknown-event debug log to swallow
these messages so OMI button presses trigger the existing _handle_button_event
handler.
| def decode_packet(self, data: bytes, strip_header: bool = True): | ||
| if len(data) <= 3: | ||
| return b'' | ||
|
|
||
| # Remove 3-byte header | ||
| if strip_header: | ||
| clean_data = bytes(data[3:]) | ||
| else: | ||
| clean_data = data |
There was a problem hiding this comment.
Length guard is applied unconditionally regardless of strip_header.
When strip_header=False, the len(data) <= 3 check still silently drops packets of 3 bytes or fewer, even though no header is being stripped. The guard should only require > 3 when the 3-byte header needs to be removed; otherwise > 0 (or even just non-empty) is sufficient.
Proposed fix
def decode_packet(self, data: bytes, strip_header: bool = True):
- if len(data) <= 3:
- return b''
-
- # Remove 3-byte header
if strip_header:
+ if len(data) <= 3:
+ return b''
clean_data = bytes(data[3:])
else:
+ if not data:
+ return b''
clean_data = data🤖 Prompt for AI Agents
In `@extras/friend-lite-sdk/friend_lite/decoder.py` around lines 8 - 16, The
length check in decode_packet is applied unconditionally causing packets to be
dropped even when strip_header is False; update decode_packet so the guard
depends on strip_header (e.g. if strip_header: require len(data) > 3 else
require len(data) > 0 or non-empty) and only return b'' when the appropriate
minimum length isn't met, then proceed to set clean_data = bytes(data[3:]) when
strip_header is True or clean_data = data when False; ensure you reference the
existing decode_packet function and the strip_header/local variable names when
making the change.
| if "notify" in char.properties: | ||
| print(" ✅ Notifiable (can stream data)") | ||
|
|
||
| asyncio.run(main()) |
There was a problem hiding this comment.
Missing if __name__ == "__main__": guard — importing this module will trigger a BLE connection.
asyncio.run(main()) executes at module level, so any import of this file (direct or transitive) will attempt a blocking BLE connection. Wrap it in a main guard.
Proposed fix
-asyncio.run(main())
+if __name__ == "__main__":
+ asyncio.run(main())📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| asyncio.run(main()) | |
| if __name__ == "__main__": | |
| asyncio.run(main()) |
🤖 Prompt for AI Agents
In `@extras/friend-lite-sdk/friend_lite/discover_characteristics.py` at line 19,
The module calls asyncio.run(main()) at import time which triggers a BLE
connection; guard that call so it only runs when executed as a script by
wrapping the invocation in if __name__ == "__main__": asyncio.run(main()).
Locate the top-level asyncio.run(main()) call in discover_characteristics.py and
move it under that main guard (keeping the existing async def main() unchanged)
so imports no longer start the BLE connection.
| async def send_audio(): | ||
| while True: | ||
| try: | ||
| chunk = await audio_queue.get() | ||
| await ws.send(chunk) | ||
| except Exception as e: | ||
| print(f"Error sending audio: {e}") | ||
| break |
There was a problem hiding this comment.
Missing None termination-signal handling in send_audio.
transcribe_wyoming (line 131) treats None from the queue as a termination signal and returns cleanly. send_audio does not — it will pass None directly to ws.send(), which will either raise or send an invalid frame. Both consumers of the same queue protocol should handle the sentinel consistently.
Proposed fix
async def send_audio():
while True:
try:
chunk = await audio_queue.get()
+ if chunk is None:
+ logger.info("Audio queue finished. Closing send_audio.")
+ return
await ws.send(chunk)
except Exception as e:
- print(f"Error sending audio: {e}")
+ logger.error(f"Error sending audio: {e}")
break🤖 Prompt for AI Agents
In `@extras/friend-lite-sdk/friend_lite/transcribe.py` around lines 43 - 50,
send_audio currently pulls from audio_queue and blindly forwards chunks to
ws.send, but transcribe_wyoming treats a None sentinel as termination; update
send_audio to check if chunk is None, break/return (closing or stopping the send
loop) instead of calling ws.send(None), and ensure any necessary cleanup (e.g.,
closing ws or signaling EOF) happens consistently with transcribe_wyoming's
termination behavior; locate the async def send_audio, the audio_queue usage and
ws.send calls to implement this sentinel handling.
| while True: # Loop for handling audio segments (and potential queue termination) | ||
| client = None | ||
| segment_has_audio = False | ||
| chunk = None # Initialize chunk to handle potential early exit | ||
|
|
||
| # 1. Connect to the server | ||
| logger.info(f"Attempting to connect to Wyoming server at {wyoming_url}") | ||
| client = AsyncClient.from_uri(wyoming_url) | ||
| await client.connect() | ||
| logger.info(f"Connected to Wyoming server at {wyoming_url}") | ||
| try: |
There was a problem hiding this comment.
Connection establishment is outside the try block — failures crash the function instead of retrying.
AsyncClient.from_uri and client.connect() on lines 97–98 execute before the try on line 100. If the Wyoming server is unreachable, ConnectionRefusedError (or similar) propagates out of the while True loop unhandled, terminating the entire coroutine. The except on line 207 and finally on line 218 only cover the inner try. This means connection failures aren't retried, and the client object created on line 97 is never disconnected on failure.
Proposed fix — move connect inside the try block
while True:
client = None
segment_has_audio = False
chunk = None
- logger.info(f"Attempting to connect to Wyoming server at {wyoming_url}")
- client = AsyncClient.from_uri(wyoming_url)
- await client.connect()
- logger.info(f"Connected to Wyoming server at {wyoming_url}")
try:
+ logger.info(f"Attempting to connect to Wyoming server at {wyoming_url}")
+ client = AsyncClient.from_uri(wyoming_url)
+ await client.connect()
+ logger.info(f"Connected to Wyoming server at {wyoming_url}")
+
# 2. Tell the server what we intend to do📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| while True: # Loop for handling audio segments (and potential queue termination) | |
| client = None | |
| segment_has_audio = False | |
| chunk = None # Initialize chunk to handle potential early exit | |
| # 1. Connect to the server | |
| logger.info(f"Attempting to connect to Wyoming server at {wyoming_url}") | |
| client = AsyncClient.from_uri(wyoming_url) | |
| await client.connect() | |
| logger.info(f"Connected to Wyoming server at {wyoming_url}") | |
| try: | |
| while True: # Loop for handling audio segments (and potential queue termination) | |
| client = None | |
| segment_has_audio = False | |
| chunk = None # Initialize chunk to handle potential early exit | |
| # 1. Connect to the server | |
| try: | |
| logger.info(f"Attempting to connect to Wyoming server at {wyoming_url}") | |
| client = AsyncClient.from_uri(wyoming_url) | |
| await client.connect() | |
| logger.info(f"Connected to Wyoming server at {wyoming_url}") | |
| # 2. Tell the server what we intend to do |
🤖 Prompt for AI Agents
In `@extras/friend-lite-sdk/friend_lite/transcribe.py` around lines 90 - 100, The
connection creation and await client.connect() happen before the inner try, so
connection failures escape the loop and the client isn't cleaned up; move the
AsyncClient.from_uri(wyoming_url) and await client.connect() inside the try
block that currently begins at the inner try (the block surrounding the rest of
the per-connection logic), and ensure the finally section always checks and
disconnects client (referencing variables AsyncClient.from_uri, client.connect,
the while True loop, and the try/except/finally) so connection errors are
caught, retried by the while loop, and any partially created client is properly
disconnected.
| async def send_button_event(button_state: str) -> None: | ||
| """Send a button event to the backend via the active WebSocket connection. | ||
|
|
||
| Args: | ||
| button_state: Button state string (e.g., "SINGLE_TAP", "DOUBLE_TAP") | ||
| """ | ||
| if _active_websocket is None: | ||
| logger.debug("No active websocket, dropping button event: %s", button_state) | ||
| return | ||
|
|
||
| event = { | ||
| "type": "button-event", | ||
| "data": {"state": button_state}, | ||
| "payload_length": None, | ||
| } | ||
| await _active_websocket.send(json.dumps(event) + "\n") | ||
| logger.info("Sent button event to backend: %s", button_state) |
There was a problem hiding this comment.
Guard button-event sends against closed WebSocket.
await _active_websocket.send(...) can raise and bubble into the caller. Catch errors and reset the shared reference to avoid cascading failures.
🔧 Suggested fix
async def send_button_event(button_state: str) -> None:
+ global _active_websocket
"""Send a button event to the backend via the active WebSocket connection.
@@
- await _active_websocket.send(json.dumps(event) + "\n")
- logger.info("Sent button event to backend: %s", button_state)
+ try:
+ await _active_websocket.send(json.dumps(event) + "\n")
+ logger.info("Sent button event to backend: %s", button_state)
+ except Exception as e:
+ logger.warning("Failed to send button event (%s): %s", button_state, e)
+ _active_websocket = None📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| async def send_button_event(button_state: str) -> None: | |
| """Send a button event to the backend via the active WebSocket connection. | |
| Args: | |
| button_state: Button state string (e.g., "SINGLE_TAP", "DOUBLE_TAP") | |
| """ | |
| if _active_websocket is None: | |
| logger.debug("No active websocket, dropping button event: %s", button_state) | |
| return | |
| event = { | |
| "type": "button-event", | |
| "data": {"state": button_state}, | |
| "payload_length": None, | |
| } | |
| await _active_websocket.send(json.dumps(event) + "\n") | |
| logger.info("Sent button event to backend: %s", button_state) | |
| async def send_button_event(button_state: str) -> None: | |
| global _active_websocket | |
| """Send a button event to the backend via the active WebSocket connection. | |
| Args: | |
| button_state: Button state string (e.g., "SINGLE_TAP", "DOUBLE_TAP") | |
| """ | |
| if _active_websocket is None: | |
| logger.debug("No active websocket, dropping button event: %s", button_state) | |
| return | |
| event = { | |
| "type": "button-event", | |
| "data": {"state": button_state}, | |
| "payload_length": None, | |
| } | |
| try: | |
| await _active_websocket.send(json.dumps(event) + "\n") | |
| logger.info("Sent button event to backend: %s", button_state) | |
| except Exception as e: | |
| logger.warning("Failed to send button event (%s): %s", button_state, e) | |
| _active_websocket = None |
🤖 Prompt for AI Agents
In `@extras/local-omi-bt/send_to_adv.py` around lines 39 - 55, The
send_button_event function currently calls await _active_websocket.send(...)
without guarding for a closed/failed socket; wrap the send in a try/except that
catches ConnectionClosed (from the websockets lib) and a general Exception, log
the error with context (include button_state and exception), and clear/reset the
shared _active_websocket reference (and close it if appropriate) so subsequent
calls won't reuse a broken socket; keep the rest of the function flow the same
and ensure the error path does not re-raise.
There was a problem hiding this comment.
Actionable comments posted: 13
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (9)
extras/speaker-recognition/src/simple_speaker_recognition/api/routers/identification.py (1)
232-238:⚠️ Potential issue | 🟠 MajorThread-safety concern:
db.similarity_thrmutation across concurrent requests is not safe.Temporarily overriding
db.similarity_thron the sharedUnifiedSpeakerDBsingleton is not safe under concurrent requests. One request's threshold mutation can interfere with another concurrent request, even within the try-finally block. Theidentify()method currently does not accept a threshold parameter, but consider refactoring it to accept one instead of relying on mutable instance state.plugins/email_summarizer/plugin.py (2)
209-209:⚠️ Potential issue | 🟠 MajorPII logged: user email addresses appear in log output.
Line 209 logs the recipient email at
INFOlevel, and Line 255 logsnotification_emailatDEBUG. Email addresses are PII and should not appear in logs to maintain GDPR/CCPA compliance. Consider masking or omitting them.Proposed fix
- logger.info(f"✅ Email summary sent to {user_email} for conversation {conversation_id}") + logger.info(f"✅ Email summary sent for conversation {conversation_id}")- logger.debug(f"Sending notification to {notification_email} for user {user_id}") + logger.debug(f"Found notification email for user {user_id}")Also applies to: 255-255
174-178:⚠️ Potential issue | 🟡 MinorUser ID leaked in
PluginResult.message.Line 177 embeds
context.user_idin the result message. If this surfaces in API responses or logs, it could be a minor information disclosure. Prefer a generic message.Proposed fix
- message=f"No email configured for user {context.user_id}" + message="No notification email configured for user"backends/advanced/src/advanced_omi_backend/client.py (1)
143-153:⚠️ Potential issue | 🟠 Major
markerslist is never reset between conversations.
start_new_conversationresetscurrent_audio_uuid,conversation_start_time,last_transcript_time, andconversation_closed, butself.markersis not cleared. Markers collected during a previous conversation will carry over and potentially be attached to the next conversation.Proposed fix
async def start_new_conversation(self): """Start a new conversation by closing current and resetting state.""" await self.close_current_conversation() # Reset conversation state self.current_audio_uuid = None self.conversation_start_time = time.time() self.last_transcript_time = None self.conversation_closed = False + self.markers = [] audio_logger.info(f"Client {self.client_id}: Started new conversation")#!/bin/bash # Check how markers are consumed downstream (e.g., in conversation_jobs) to confirm they should be cleared per-conversation rg -n --type=py -C3 'markers' --glob '!**/node_modules/**' --glob '!**/friend-lite-sdk/**' | head -80backends/advanced/src/advanced_omi_backend/services/transcription/__init__.py (1)
220-233:⚠️ Potential issue | 🟡 MinorUnreachable dead code after
raiseinexceptblock.Lines 227–232 are unreachable because the
except httpx.HTTPStatusErrorblock always raises on line 222. This debug logging will never execute.🔧 Suggested fix — move debug logging before the error handling or remove it
except httpx.HTTPStatusError as e: status = e.response.status_code raise RuntimeError( f"Transcription service '{self._name}' at {url} returned HTTP {status}. " f"{'Check your API key.' if status in (401, 403) else ''}" ) from e - - # DEBUG: Log Deepgram response structure - if "results" in data and "channels" in data.get("results", {}): - channels = data["results"]["channels"] - if channels and "alternatives" in channels[0]: - alt = channels[0]["alternatives"][0] - logger.debug(f"DEBUG Registry: Deepgram alternative keys: {list(alt.keys())}")backends/advanced/Docs/plugin-development-guide.md (1)
362-367:⚠️ Potential issue | 🟡 MinorExample uses
datetime.utcnow()without importingdatetime.The
TodoExtractorPluginexample callsdatetime.utcnow()on line 366, butdatetimeis not in the imports shown at lines 297-301. This will confuse developers copying the example.backends/advanced/src/advanced_omi_backend/controllers/audio_controller.py (1)
145-145:⚠️ Potential issue | 🔴 Critical
NoneTypecrash:file.filename.rsplit(...)whenfile.filename is None.Line 145 calls
.rsplit()onfile.filenamebefore theif file.filenameguard evaluates. Iffile.filenameisNone, this raisesAttributeError. Use the safefilenamelocal variable instead.🐛 Proposed fix
- title = file.filename.rsplit('.', 1)[0][:50] if file.filename else "Uploaded Audio" + title = filename.rsplit('.', 1)[0][:50] if filename != "unknown" else "Uploaded Audio"backends/advanced/src/advanced_omi_backend/workers/conversation_jobs.py (1)
661-710:⚠️ Potential issue | 🟠 MajorDuplicate code block: streaming transcription completion wait is copy-pasted.
Lines 661-686 and lines 688-710 are identical — both wait for
transcription:complete:{session_id}. This appears to be a copy-paste mistake that doubles the maximum wait time from 30s to 60s.🐛 Remove the duplicate block
if waited_streaming >= max_wait_streaming: logger.warning( f"⚠️ Timed out waiting for streaming completion signal for {session_id} " f"(waited {max_wait_streaming}s), proceeding with available transcript" ) - # Wait for streaming transcription consumer to complete before reading transcript - # This fixes the race condition where conversation job reads transcript before - # streaming consumer stores all final results (seen as 24+ second delay in logs) - completion_key = f"transcription:complete:{session_id}" - max_wait_streaming = 30 # seconds - waited_streaming = 0.0 - while waited_streaming < max_wait_streaming: - completion_status = await redis_client.get(completion_key) - if completion_status: - status_str = completion_status.decode() if isinstance(completion_status, bytes) else completion_status - if status_str == "error": - logger.warning(f"⚠️ Streaming transcription ended with error for {session_id}, proceeding anyway") - else: - logger.info(f"✅ Streaming transcription confirmed complete for {session_id}") - break - await asyncio.sleep(0.5) - waited_streaming += 0.5 - - if waited_streaming >= max_wait_streaming: - logger.warning( - f"⚠️ Timed out waiting for streaming completion signal for {session_id} " - f"(waited {max_wait_streaming}s), proceeding with available transcript" - ) - # Wait for audio_streaming_persistence_job to complete and write MongoDB chunksbackends/advanced/webui/src/pages/Conversations.tsx (1)
276-284:⚠️ Potential issue | 🟡 MinorDouble-fetch of conversations on mount.
The
useEffect(…, [])on Line 276 callsloadConversations(), and theuseEffect(…, [debugMode])on Line 282 also fires on mount with the initialdebugModevalue. This means conversations are fetched twice on every page load.Consider removing the
loadConversations()call from the first effect (keepingloadEnrolledSpeakers()there) and letting the[debugMode]effect be the sole trigger:Proposed fix
useEffect(() => { - loadConversations() loadEnrolledSpeakers() }, []) // Reload conversations when debug mode toggles (to include/exclude orphans) useEffect(() => { loadConversations() }, [debugMode])
🤖 Fix all issues with AI agents
In
`@backends/advanced/src/advanced_omi_backend/controllers/conversation_controller.py`:
- Around line 470-490: The reprocess_orphan function currently allows
reprocessing any conversation with audio chunks; add an explicit orphan guard
early (before checking AudioChunkDocument) that validates conversation is
actually an orphan by checking its relevant fields (e.g., ensure
conversation.always_persist is False and that conversation.processing_status
and/or conversation.deletion_reason indicate a deleted/orphaned state—or use an
existing conversation.is_orphan flag if present); if the conversation does not
meet the orphan criteria, return a 400/403 JSONResponse with an explanatory
error; place this check after loading the Conversation in reprocess_orphan and
before counting AudioChunkDocument items so non-orphan conversations cannot be
reprocessed.
In
`@backends/advanced/src/advanced_omi_backend/controllers/session_controller.py`:
- Around line 68-95: Replace the non-atomic exists→hset in
request_conversation_close with an atomic Redis operation: use a Lua EVAL script
(or Redis transaction with WATCH/MULTI/EXEC) that checks for the session key
(session_key) and only sets the "conversation_close_requested" field to the
provided reason if the key still exists, returning a boolean result; update
request_conversation_close to call this atomic script, return True only when the
script set the field, return False when the session was missing, and only emit
the logger.info message when the atomic set succeeded.
In `@backends/advanced/src/advanced_omi_backend/utils/audio_utils.py`:
- Around line 145-152: The FFmpeg subprocess call in convert_any_to_wav (and
similarly in resample_audio_with_ffmpeg) lacks a timeout; wrap the await
process.communicate(input=file_data) in asyncio.wait_for with a sane timeout
value, catch asyncio.TimeoutError, then terminate/kill the process (call
process.terminate() or process.kill()) and await process.wait() to ensure
cleanup, capture any available stdout/stderr, and raise/return a clear timeout
error so the async worker isn't blocked indefinitely.
In `@backends/advanced/webui/src/contexts/RecordingContext.tsx`:
- Around line 96-111: In refreshDevices, after computing and calling
setAvailableDevices(audioInputs), check whether the current selectedDeviceId
(state variable) still exists in the new audioInputs list and if not,
clear/reset it (e.g., setSelectedDeviceId(undefined/null/''), depending on
existing state typing) so the UI dropdown doesn't show a blank choice and future
getUserMedia calls that use { deviceId: { exact: selectedDeviceId } } won't
throw OverconstrainedError; update the logic in the refreshDevices callback
(used by useEffect) to perform this validation and reset selectedDeviceId when
the selected device is absent.
- Around line 479-483: After creating AudioContext in RecordingContext
(audioContextRef.current = audioContext in the AudioContext({ sampleRate: 16000
}) block), check audioContext.sampleRate and if it !== 16000, fail-fast or
perform client-side resampling before sending audio to the backend: log a clear
warning, close the created AudioContext if necessary, then either (a) create an
OfflineAudioContext or use an AudioWorklet/ScriptProcessor to resample to 16 kHz
and set a flag to mark streams as 16k, or (b) reject/start-stream with an
explicit error so the backend won’t receive mismatched-rate audio; ensure the
audio-start message and downstream processing use the actual output sample rate
(16k) after resampling and reference audioContextRef/currentAudioSampleRate and
the code that sends the audio-start WebSocket message to implement this guard.
In `@backends/advanced/webui/src/pages/Queue.tsx`:
- Around line 2051-2156: The Events UI assumes evt.plugins_executed and
evt.user_id always exist which can throw when absent; update the rendering
inside the filtered.map callback (the code that references
evt.plugins_executed.length, .map, .filter and evt.user_id.length/slice) to use
defensive defaults and optional chaining—treat plugins_executed as
(evt.plugins_executed || []) and user_id as (evt.user_id || '') (or use
evt.plugins_executed?.length and evt.plugins_executed?.map with fallbacks) so
.length/.map/.filter calls never run on undefined and the UI shows sensible
defaults like "none" or truncated empty id instead of crashing.
In `@backends/advanced/webui/src/pages/Upload.tsx`:
- Around line 6-7: The videoWarning state is only cleared in clearCompleted so
it persists when users remove video files individually; update removeFile (and
any other places that mutate the files list, e.g., where files are
added/cleared) to recompute videoWarning by checking the remaining files'
extensions against VIDEO_EXTENSIONS (using SUPPORTED_EXTENSIONS/VIDEO_EXTENSIONS
for identification) and set videoWarning to false when no remaining file matches
a video extension; ensure the same recalculation logic is applied after
removeFile and after any list mutations so videoWarning always reflects the
current files.
In `@extras/friend-lite-sdk/friend_lite/transcribe.py`:
- Line 155: Remove the unnecessary f-string prefix on the logger call: replace
the logger.debug call that currently reads logger.debug(f"Wyoming: Sending
AudioStop...") with a normal string literal so it becomes logger.debug("Wyoming:
Sending AudioStop..."); this change targets the logger.debug invocation in
friend_lite/transcribe.py (the "Wyoming: Sending AudioStop..." debug message)
and removes the unused f-prefix.
In `@extras/friend-lite-sdk/LICENSE`:
- Line 3: Update the copyright year in the LICENSE file from "2024" to the
correct year for this derivative work (e.g., "2026"); locate the line containing
"Copyright (c) 2024 Chronicle AI Contributors" and replace the year so the
LICENSE reflects the actual creation year of this addition.
In `@extras/friend-lite-sdk/README.md`:
- Around line 22-30: The example imports ButtonState and parse_button_event but
doesn't use them; either remove them from the import and keep only
OmiConnection, or update the example to demonstrate button handling by calling
the connection's button subscription (e.g., using a subscribe_button or similar
callback), passing events into parse_button_event and checking ButtonState to
show handling alongside the existing subscribe_audio and wait_until_disconnected
calls; ensure you reference OmiConnection, parse_button_event, and ButtonState
in the updated snippet so no imported symbols are unused.
In `@extras/local-omi-bt/connect-omi.py`:
- Around line 74-78: The created Task from
loop.create_task(send_button_event(state.name)) is immediately discarded and may
be GC'd; keep a strong reference and remove it when done. Replace the discard
with storing the Task in a module- or instance-level collection (e.g., a
set/list named _pending_tasks), create the Task via
loop.create_task(send_button_event(...)), add a done callback to remove it from
_pending_tasks and log exceptions if task.exception() is set; reference
send_button_event and loop.create_task when making these edits.
In `@extras/local-omi-bt/requirements.txt`:
- Line 5: Replace the non-editable file URL dependency "friend-lite-sdk @
file:../friend-lite-sdk" in the requirements entry with an editable install as
recommended by the SDK docs: use "-e ../friend-lite-sdk" to enable
development/editable installs for friend-lite-sdk; if you intentionally prefer a
non-editable install, add a short comment in the requirements (or README)
explaining why the file: reference was chosen instead of the editable "-e" form.
In `@extras/local-omi-bt/scan_devices.py`:
- Around line 17-19: The device name filter in scan_devices.py (the any(...)
check that appends to neo_devices based on d.name) is inconsistent with
connect-omi.py; change the predicate used in the any(...) call so both scripts
use the same token set (e.g., include "omi" and "neo" rather than "neo" and
"sapien") so that devices like "OMI-xxx" and "neosapien-xxx" are discovered
consistently; update the any(...) token list inside the d.name check where
neo_devices.append(d) is called (or centralize the token list into a shared
constant if preferred) to match connect-omi.py’s filter.
🧹 Nitpick comments (21)
backends/advanced/webui/src/pages/MemoryDetail.tsx (1)
176-178: Unreachableelsebranch — dead code.All members of the union type (
undefined | null→ line 161,number→ line 167,string→ line 169) are already handled before this point, sodateInputisneverhere. This branch can never execute.🧹 Suggested removal
- } else { - date = new Date(dateInput) - } + }You'll need to add a
dateinitializer or restructure slightly to avoid a "possibly unassigned" error, e.g., by assigning a default at thelet datedeclaration:- let date: Date + let date: Date = new Date(NaN) // fallback; caught by isNaN check belowbackends/advanced/webui/src/pages/LiveRecord.tsx (1)
57-83: Microphone selector looks good overall.Clean implementation with proper disabled state during recording and a sensible fallback label for unlabeled devices. Two minor notes:
- Accessibility: The
<label>isn't associated with the<select>viahtmlFor/id. Screen readers won't link them.- Stale selection: If the selected device is physically disconnected,
selectedDeviceIdmay reference a device no longer inavailableDevices, causing the<select>to show a blank value. Consider resettingselectedDeviceIdtonullinsideRecordingContextwhen the device list changes and the selected ID is no longer present.Accessibility fix
- <label className="text-sm font-medium text-gray-700 dark:text-gray-300 flex-shrink-0"> + <label htmlFor="mic-select" className="text-sm font-medium text-gray-700 dark:text-gray-300 flex-shrink-0"> Microphone: </label> <select + id="mic-select" value={recording.selectedDeviceId ?? ''}extras/friend-lite-sdk/pyproject.toml (1)
20-20:wyomingdependency has no version pin.Unlike all other dependencies,
wyominghas no minimum version constraint. This could lead to breakage if the package introduces incompatible changes.Pin a minimum version
-wyoming = ["wyoming"] +wyoming = ["wyoming>=1.5.0"](Replace
1.5.0with whatever version you've tested against.)extras/friend-lite-sdk/friend_lite/button.py (1)
16-24: Consider handling unknown button state values gracefully.
ButtonState(state)will raise aValueErrorif the device sends an unrecognized state value (e.g., a firmware update adds state6). If robustness against device firmware changes is desired, consider returning a fallback or raising a more descriptive error.Example: graceful fallback
- return ButtonState(state) + try: + return ButtonState(state) + except ValueError: + raise ValueError(f"Unknown button state: {state}")backends/advanced/src/advanced_omi_backend/models/conversation.py (1)
126-130:markersusesList[Dict[str, Any]]— consider a typed model.An untyped dict list makes it easy to persist inconsistent marker shapes. Even a lightweight
Markermodel (e.g., withtype,timestamp, and optionalmetadatafields) would enforce a contract and improve downstream consumers' experience.This is fine for a draft, but worth tightening before merge.
extras/friend-lite-sdk/friend_lite/discover_characteristics.py (1)
4-4: Hardcoded placeholder MAC address should use a CLI argument or env var.The hardcoded
OMI_MACwith a comment "Replace with your actual MAC" is fragile. For a utility script in an SDK, accept the address viasys.argvor an environment variable to avoid accidental commits of real device addresses.Example
-OMI_MAC = "7F52EC55-50C9-D1B9-E8D7-19B83217C97D" # Replace with your actual MAC +import sys + +OMI_MAC = sys.argv[1] if len(sys.argv) > 1 else os.environ.get("OMI_MAC", "") +if not OMI_MAC: + print("Usage: python discover_characteristics.py <MAC_ADDRESS>") + sys.exit(1)extras/friend-lite-sdk/friend_lite/decoder.py (1)
22-23: Uselogginginstead ofThis SDK module uses
Proposed fix
+import logging from opuslib import Decoder +logger = logging.getLogger(__name__) + class OmiOpusDecoder: def __init__(self): @@ -20,5 +23,5 @@ try: pcm = self.decoder.decode(clean_data, 960, decode_fec=False) return pcm except Exception as e: - print("Opus decode error:", e) + logger.warning("Opus decode error: %s", e) return b''extras/friend-lite-sdk/friend_lite/bluetooth.py (1)
9-12:print_devicescallsasyncio.run()— will crash if called from an existing event loop.If called from an already-running async context (e.g., Jupyter, or another async framework),
asyncio.run()raisesRuntimeError. Consider making thisasyncor using a loop-aware approach.extras/friend-lite-sdk/friend_lite/transcribe.py (2)
32-83:transcribe()usesprint()throughout instead of the module-levellogger.A
loggeris already defined at Line 23 and used consistently intranscribe_wyoming. The Deepgramtranscribe()function should use it as well for consistent log management.
33-33: API key is passed via URL header, but the Deepgram URL has hardcoded query parameters.The model (
nova), language (en-US), encoding, sample rate, and channels are all hardcoded in the URL string. Consider parameterizing at least the model and language to make this function reusable.backends/advanced/webui/src/pages/Upload.tsx (1)
112-117: Stale closure inhandleDrop—handleFileSelectis captured once.
handleDropusesuseCallbackwith an empty dependency array but callshandleFileSelect, which is redefined on every render. The captured reference will be the initial one. In this case it works by accident becausehandleFileSelectdoesn't read any state, but this is fragile. Consider addinghandleFileSelectas a dependency or wrapping it inuseCallbacktoo.backends/advanced/Docs/plugin-development-guide.md (1)
216-236: Add a language identifier to the fenced code block.The data flow diagram block (line 216) lacks a language identifier. Use
```textto satisfy linting and signal intent.extras/local-omi-bt/connect-omi.py (1)
148-166: RedundantBleakClientusage and nested retry logic is deeply coupled.The initial connection check at line 149 and the fallback re-scan + re-connect at lines 153-166 create a deeply nested try/except structure. The
test_clientvariable is also unused (Ruff F841). Consider extracting a helper likeverify_or_pick_device(mac) -> strto flatten this.backends/advanced/src/advanced_omi_backend/plugins/base.py (1)
21-21: Consider a forward reference orTYPE_CHECKINGimport forservicestype.
Optional[Any]loses all type information. SincePluginServiceslives in the same package, aTYPE_CHECKINGguard would give IDE support without circular import risk.💡 Suggested improvement
+from typing import TYPE_CHECKING +if TYPE_CHECKING: + from advanced_omi_backend.plugins.services import PluginServices + `@dataclass` class PluginContext: """Context passed to plugin execution""" user_id: str event: str data: Dict[str, Any] metadata: Dict[str, Any] = field(default_factory=dict) - services: Optional[Any] = None + services: Optional["PluginServices"] = Noneplugins/test_button_actions/plugin.py (2)
22-22: Annotate mutable class attribute withClassVar.Ruff RUF012 flags this correctly. Mutable class-level defaults should use
ClassVarto signal they're not instance attributes.Suggested fix
+from typing import Any, ClassVar, Dict, List, Optional -from typing import Any, Dict, List, Optional ... - SUPPORTED_ACCESS_LEVELS: List[str] = ["button"] + SUPPORTED_ACCESS_LEVELS: ClassVar[List[str]] = ["button"]
74-101: Unusedaction_configparameter — consider using it for a configurable close reason.
action_configis accepted but never read. SinceConversationCloseReasonhas multiple values, the close reason could be driven by the config (e.g.,action_config.get("reason", "button_close")). If that flexibility isn't needed, drop the parameter to match_handle_call_plugin's pattern whereaction_configis used.Option A: use action_config for the reason
async def _handle_close_conversation( self, context: PluginContext, action_config: dict ) -> PluginResult: ... + reason_str = action_config.get("reason", ConversationCloseReason.BUTTON_CLOSE.value) + try: + reason = ConversationCloseReason(reason_str) + except ValueError: + reason = ConversationCloseReason.BUTTON_CLOSE + success = await context.services.close_conversation( session_id=session_id, - reason=ConversationCloseReason.BUTTON_CLOSE, + reason=reason, )Option B: remove unused parameter
async def _handle_close_conversation( - self, context: PluginContext, action_config: dict + self, context: PluginContext ) -> PluginResult:(and update the call site on line 68 accordingly)
backends/advanced/src/advanced_omi_backend/plugins/router.py (2)
104-110: Sync Redis client used from asyncdispatch_eventpath.
_log_eventuses a synchronousredis.from_urlclient, but it's called from the asyncdispatch_eventmethod. Thepipe.execute()call on line 313 will block the event loop. For a local Redis lpush+ltrim this is likely sub-millisecond, but under network issues or Redis load it could stall the async loop.Consider either: (a) accepting this as-is since it's non-critical logging, or (b) wrapping the call in
asyncio.to_thread/ usingaioredisto be fully non-blocking.
273-288: VerifyPluginEventenum comparison with stringeventparameter.
dispatch_eventreceiveseventasstr, but_execute_plugincompares it withPluginEventenum members usingin. This works becausePluginEvent(str, Enum)means each enum member compares equal to its string value (e.g.,"button.single_press" == PluginEvent.BUTTON_SINGLE_PRESSisTrue). The comment on line 274 correctly documents this.One edge case:
event in (PluginEvent.CONVERSATION_COMPLETE,)andevent in (PluginEvent.MEMORY_PROCESSED,)— single-element tuples are fine but slightly unusual. If no additional events will be added to those groups, a direct==comparison reads more clearly.backends/advanced/src/advanced_omi_backend/services/plugin_service.py (1)
28-41:parents[5]is fragile — breaks if the file is relocated.The hardcoded ancestor index assumes the exact directory depth of
plugin_service.py. If the file is ever moved, this silently resolves to the wrong path.Consider a more robust approach, such as searching upward for a known marker file (e.g.,
pyproject.toml,.git, or a sentinel) or relying solely on the env var / Docker path.More robust alternative
# Local dev: plugin_service.py is at <repo>/backends/advanced/src/advanced_omi_backend/services/ - repo_root = Path(__file__).resolve().parents[5] - return repo_root / "plugins" + # Walk up looking for a repo root marker + candidate = Path(__file__).resolve().parent + while candidate != candidate.parent: + if (candidate / "plugins").is_dir(): + return candidate / "plugins" + candidate = candidate.parent + # Absolute fallback + return Path(__file__).resolve().parents[5] / "plugins"backends/advanced/src/advanced_omi_backend/plugins/services.py (1)
44-54: New Redis connection created perclose_conversationcall.Each invocation opens a fresh Redis connection, uses it for a single
hset, then closes it. If this method is called frequently, consider accepting a shared Redis client or connection pool via the constructor instead.backends/advanced/src/advanced_omi_backend/controllers/conversation_controller.py (1)
79-81: Redis URL resolved viaos.getenvon every call.Same pattern as
PluginServices— a new Redis connection is created per invocation. Consider centralizing the Redis URL resolution or passing a shared client/pool to avoid inconsistent defaults and connection churn.
| async def reprocess_orphan(conversation_id: str, user: User): | ||
| """Reprocess an orphan audio session - restore if deleted and enqueue full processing chain.""" | ||
| try: | ||
| conversation = await Conversation.find_one(Conversation.conversation_id == conversation_id) | ||
| if not conversation: | ||
| return JSONResponse(status_code=404, content={"error": "Conversation not found"}) | ||
|
|
||
| # Check ownership | ||
| if not user.is_superuser and conversation.user_id != str(user.user_id): | ||
| return JSONResponse(status_code=403, content={"error": "Access forbidden"}) | ||
|
|
||
| # Verify audio chunks exist (check both deleted and non-deleted) | ||
| total_chunks = await AudioChunkDocument.find( | ||
| AudioChunkDocument.conversation_id == conversation_id | ||
| ).count() | ||
|
|
||
| if total_chunks == 0: | ||
| return JSONResponse( | ||
| status_code=400, | ||
| content={"error": "No audio data found for this conversation"}, | ||
| ) |
There was a problem hiding this comment.
reprocess_orphan doesn't validate that the conversation is actually an orphan.
Any conversation with audio chunks can be reprocessed through this endpoint regardless of its always_persist, processing_status, or deletion_reason values. This could lead to unintended reprocessing of normal conversations. Consider adding a guard:
Sketch
+ # Verify this is actually an orphan conversation
+ is_orphan = (
+ (conversation.always_persist and conversation.processing_status in
+ ("pending_transcription", "transcription_failed"))
+ or (conversation.deleted and conversation.deletion_reason in
+ ("no_meaningful_speech", "audio_file_not_ready", "no_meaningful_speech_batch_transcription"))
+ )
+ if not is_orphan:
+ return JSONResponse(
+ status_code=400,
+ content={"error": "Conversation is not an orphan audio session"},
+ )📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| async def reprocess_orphan(conversation_id: str, user: User): | |
| """Reprocess an orphan audio session - restore if deleted and enqueue full processing chain.""" | |
| try: | |
| conversation = await Conversation.find_one(Conversation.conversation_id == conversation_id) | |
| if not conversation: | |
| return JSONResponse(status_code=404, content={"error": "Conversation not found"}) | |
| # Check ownership | |
| if not user.is_superuser and conversation.user_id != str(user.user_id): | |
| return JSONResponse(status_code=403, content={"error": "Access forbidden"}) | |
| # Verify audio chunks exist (check both deleted and non-deleted) | |
| total_chunks = await AudioChunkDocument.find( | |
| AudioChunkDocument.conversation_id == conversation_id | |
| ).count() | |
| if total_chunks == 0: | |
| return JSONResponse( | |
| status_code=400, | |
| content={"error": "No audio data found for this conversation"}, | |
| ) | |
| async def reprocess_orphan(conversation_id: str, user: User): | |
| """Reprocess an orphan audio session - restore if deleted and enqueue full processing chain.""" | |
| try: | |
| conversation = await Conversation.find_one(Conversation.conversation_id == conversation_id) | |
| if not conversation: | |
| return JSONResponse(status_code=404, content={"error": "Conversation not found"}) | |
| # Check ownership | |
| if not user.is_superuser and conversation.user_id != str(user.user_id): | |
| return JSONResponse(status_code=403, content={"error": "Access forbidden"}) | |
| # Verify audio chunks exist (check both deleted and non-deleted) | |
| total_chunks = await AudioChunkDocument.find( | |
| AudioChunkDocument.conversation_id == conversation_id | |
| ).count() | |
| if total_chunks == 0: | |
| return JSONResponse( | |
| status_code=400, | |
| content={"error": "No audio data found for this conversation"}, | |
| ) | |
| # Verify this is actually an orphan conversation | |
| is_orphan = ( | |
| (conversation.always_persist and conversation.processing_status in | |
| ("pending_transcription", "transcription_failed")) | |
| or (conversation.deleted and conversation.deletion_reason in | |
| ("no_meaningful_speech", "audio_file_not_ready", "no_meaningful_speech_batch_transcription")) | |
| ) | |
| if not is_orphan: | |
| return JSONResponse( | |
| status_code=400, | |
| content={"error": "Conversation is not an orphan audio session"}, | |
| ) |
🤖 Prompt for AI Agents
In
`@backends/advanced/src/advanced_omi_backend/controllers/conversation_controller.py`
around lines 470 - 490, The reprocess_orphan function currently allows
reprocessing any conversation with audio chunks; add an explicit orphan guard
early (before checking AudioChunkDocument) that validates conversation is
actually an orphan by checking its relevant fields (e.g., ensure
conversation.always_persist is False and that conversation.processing_status
and/or conversation.deletion_reason indicate a deleted/orphaned state—or use an
existing conversation.is_orphan flag if present); if the conversation does not
meet the orphan criteria, return a 400/403 JSONResponse with an explanatory
error; place this check after loading the Conversation in reprocess_orphan and
before counting AudioChunkDocument items so non-orphan conversations cannot be
reprocessed.
backends/advanced/src/advanced_omi_backend/controllers/session_controller.py
Show resolved
Hide resolved
| process = await asyncio.create_subprocess_exec( | ||
| *cmd, | ||
| stdin=asyncio.subprocess.PIPE, | ||
| stdout=asyncio.subprocess.PIPE, | ||
| stderr=asyncio.subprocess.PIPE, | ||
| ) | ||
|
|
||
| stdout, stderr = await process.communicate(input=file_data) |
There was a problem hiding this comment.
No timeout on FFmpeg subprocess — large or malformed files could hang indefinitely.
process.communicate() has no timeout. A malformed file could cause FFmpeg to hang, blocking the async worker. The same pattern exists in resample_audio_with_ffmpeg above, but convert_any_to_wav handles untrusted user uploads, making it higher risk.
💡 Add a timeout
- stdout, stderr = await process.communicate(input=file_data)
+ try:
+ stdout, stderr = await asyncio.wait_for(
+ process.communicate(input=file_data), timeout=300 # 5 minutes
+ )
+ except asyncio.TimeoutError:
+ process.kill()
+ raise AudioValidationError(f"FFmpeg conversion timed out for {ext} file")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| process = await asyncio.create_subprocess_exec( | |
| *cmd, | |
| stdin=asyncio.subprocess.PIPE, | |
| stdout=asyncio.subprocess.PIPE, | |
| stderr=asyncio.subprocess.PIPE, | |
| ) | |
| stdout, stderr = await process.communicate(input=file_data) | |
| process = await asyncio.create_subprocess_exec( | |
| *cmd, | |
| stdin=asyncio.subprocess.PIPE, | |
| stdout=asyncio.subprocess.PIPE, | |
| stderr=asyncio.subprocess.PIPE, | |
| ) | |
| try: | |
| stdout, stderr = await asyncio.wait_for( | |
| process.communicate(input=file_data), timeout=300 # 5 minutes | |
| ) | |
| except asyncio.TimeoutError: | |
| process.kill() | |
| raise AudioValidationError(f"FFmpeg conversion timed out for {ext} file") |
🤖 Prompt for AI Agents
In `@backends/advanced/src/advanced_omi_backend/utils/audio_utils.py` around lines
145 - 152, The FFmpeg subprocess call in convert_any_to_wav (and similarly in
resample_audio_with_ffmpeg) lacks a timeout; wrap the await
process.communicate(input=file_data) in asyncio.wait_for with a sane timeout
value, catch asyncio.TimeoutError, then terminate/kill the process (call
process.terminate() or process.kill()) and await process.wait() to ensure
cleanup, capture any available stdout/stderr, and raise/return a clear timeout
error so the async worker isn't blocked indefinitely.
| const refreshDevices = useCallback(async () => { | ||
| try { | ||
| const devices = await navigator.mediaDevices.enumerateDevices() | ||
| const audioInputs = devices.filter(d => d.kind === 'audioinput') | ||
| setAvailableDevices(audioInputs) | ||
| } catch (e) { | ||
| console.warn('Failed to enumerate audio devices:', e) | ||
| } | ||
| }, []) | ||
|
|
||
| // Initial device enumeration + listen for device changes | ||
| useEffect(() => { | ||
| refreshDevices() | ||
| navigator.mediaDevices.addEventListener('devicechange', refreshDevices) | ||
| return () => navigator.mediaDevices.removeEventListener('devicechange', refreshDevices) | ||
| }, [refreshDevices]) |
There was a problem hiding this comment.
Reset selectedDeviceId when the selected device disappears from the list.
When a device is disconnected, refreshDevices updates availableDevices but selectedDeviceId may still reference a removed device. This causes the UI dropdown to show a blank selection and, more importantly, the next getUserMedia call will fail with an OverconstrainedError due to { exact: selectedDeviceId }.
Suggested fix
const refreshDevices = useCallback(async () => {
try {
const devices = await navigator.mediaDevices.enumerateDevices()
const audioInputs = devices.filter(d => d.kind === 'audioinput')
setAvailableDevices(audioInputs)
+ setSelectedDeviceId(prev => {
+ if (prev && !audioInputs.some(d => d.deviceId === prev)) {
+ console.log('🎤 Selected device removed, resetting to default')
+ return null
+ }
+ return prev
+ })
} catch (e) {
console.warn('Failed to enumerate audio devices:', e)
}
}, [])🤖 Prompt for AI Agents
In `@backends/advanced/webui/src/contexts/RecordingContext.tsx` around lines 96 -
111, In refreshDevices, after computing and calling
setAvailableDevices(audioInputs), check whether the current selectedDeviceId
(state variable) still exists in the new audioInputs list and if not,
clear/reset it (e.g., setSelectedDeviceId(undefined/null/''), depending on
existing state typing) so the UI dropdown doesn't show a blank choice and future
getUserMedia calls that use { deviceId: { exact: selectedDeviceId } } won't
throw OverconstrainedError; update the logic in the refreshDevices callback
(used by useEffect) to perform this validation and reset selectedDeviceId when
the selected device is absent.
| // Create AudioContext at 16kHz to match the backend pipeline expectation. | ||
| // The browser will internally resample from the mic's native rate (e.g. 48kHz). | ||
| const audioContext = new AudioContext({ sampleRate: 16000 }) | ||
| audioContextRef.current = audioContext | ||
| console.log(`🎧 AudioContext created, sample rate: ${audioContext.sampleRate}Hz`) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Search backend for sample rate expectations and audio processing
find . -type f \( -name "*.py" \) ! -path "*/test*" ! -path "*/.venv/*" ! -path "*/node_modules/*" | head -20Repository: SimpleOpenSoftware/chronicle
Length of output: 1227
🏁 Script executed:
#!/bin/bash
# Look for audio processing and sample rate handling in backend
rg -n "16000|16_000|sample.?rate|resample" --type=py -g '!*test*' -g '!.venv' | grep -i "audio\|transcri\|process\|rate" | head -100Repository: SimpleOpenSoftware/chronicle
Length of output: 12115
🏁 Script executed:
#!/bin/bash
# Find audio processing or transcription pipeline files
fd -t f "audio|transcrip|pipeline" . --type f | grep -E "\.(py)$" | head -20Repository: SimpleOpenSoftware/chronicle
Length of output: 1505
🏁 Script executed:
#!/bin/bash
# Check the main audio service and transcription pipeline for sample rate handling
cat -n ./backends/advanced/src/advanced_omi_backend/services/audio_service.py | head -150Repository: SimpleOpenSoftware/chronicle
Length of output: 6488
🏁 Script executed:
#!/bin/bash
# Check transcription jobs for sample rate expectations
cat -n ./backends/advanced/src/advanced_omi_backend/workers/transcription_jobs.py | head -200Repository: SimpleOpenSoftware/chronicle
Length of output: 8491
🏁 Script executed:
#!/bin/bash
# Check audio utilities for sample rate conversions or validations
cat -n ./backends/advanced/src/advanced_omi_backend/utils/audio_utils.pyRepository: SimpleOpenSoftware/chronicle
Length of output: 19917
🏁 Script executed:
#!/bin/bash
# Verify if transcription services handle variable sample rates or expect 16kHz
rg -n "transcribe|speech_to_text" backends/advanced/src/advanced_omi_backend/workers/transcription_jobs.py -A 5 | head -60Repository: SimpleOpenSoftware/chronicle
Length of output: 1580
🏁 Script executed:
#!/bin/bash
# Check how the reconstructed audio is actually used in transcription
rg -n "reconstruct_wav_from_conversation" backends/advanced/src/advanced_omi_backend/workers/transcription_jobs.py -B 2 -A 8Repository: SimpleOpenSoftware/chronicle
Length of output: 920
🏁 Script executed:
#!/bin/bash
# Find the WebSocket audio chunk handler to see how it processes incoming audio
rg -n "audio_chunk|AudioChunk" backends/advanced/src/advanced_omi_backend/routers/modules/audio_routes.py -B 3 -A 3 | head -100Repository: SimpleOpenSoftware/chronicle
Length of output: 2823
🏁 Script executed:
#!/bin/bash
# Check WebSocket endpoint handlers in controllers
rg -n "websocket|audio" backends/advanced/src/advanced_omi_backend/controllers/audio_controller.py -B 2 -A 5 | head -80Repository: SimpleOpenSoftware/chronicle
Length of output: 2988
🏁 Script executed:
#!/bin/bash
# Find WebSocket endpoint for real-time audio streaming
find backends/advanced/src -name "*.py" -type f | xargs rg -l "websocket|WebSocket" | head -10Repository: SimpleOpenSoftware/chronicle
Length of output: 805
🏁 Script executed:
#!/bin/bash
# Look for the actual WebSocket handler receiving audio from browser
rg -n "def.*websocket|@.*websocket" backends/advanced/src -A 20 | head -150Repository: SimpleOpenSoftware/chronicle
Length of output: 17077
🏁 Script executed:
#!/bin/bash
# Get the full WebSocket endpoint implementation to see how audio chunks are handled
cat -n backends/advanced/src/advanced_omi_backend/routers/modules/websocket_routes.py | head -200Repository: SimpleOpenSoftware/chronicle
Length of output: 1868
🏁 Script executed:
#!/bin/bash
# Check RecordingContext to see if there's any rate adjustment or validation on the browser side
cat -n backends/advanced/webui/src/contexts/RecordingContext.tsx | sed -n '470,500p'Repository: SimpleOpenSoftware/chronicle
Length of output: 1464
🏁 Script executed:
#!/bin/bash
# Check the pcm_websocket handler to see if it validates or enforces sample rate
rg -n "handle_pcm_websocket" backends/advanced/src/advanced_omi_backend/controllers/websocket_controller.py -A 50 | head -100Repository: SimpleOpenSoftware/chronicle
Length of output: 2699
🏁 Script executed:
#!/bin/bash
# Search for sendAudioStartMessage in RecordingContext to see what rate information is sent to backend
rg -n "sendAudioStartMessage\|audio-start" backends/advanced/webui/src/contexts/RecordingContext.tsx -B 3 -A 8Repository: SimpleOpenSoftware/chronicle
Length of output: 54
🏁 Script executed:
#!/bin/bash
# Search for the sendAudioStartMessage function definition
rg -n "sendAudioStartMessage" backends/advanced/webui/src/contexts/RecordingContext.tsx -B 2 -A 15Repository: SimpleOpenSoftware/chronicle
Length of output: 1977
🏁 Script executed:
#!/bin/bash
# Look for where audio-start message is constructed with sample rate
rg -n "audio-start\|audioStartMessage" backends/advanced/webui/src/contexts/RecordingContext.tsx -B 5 -A 10Repository: SimpleOpenSoftware/chronicle
Length of output: 54
Browser may not honor the 16 kHz sampleRate constraint — verify with backend expectations.
AudioContext({ sampleRate: 16000 }) is a hint; the spec allows the browser to use a different rate (commonly 44100 or 48000). The code correctly reads back audioContext.sampleRate and sends it to the backend in the audio-start message. However, the WebSocket streaming path is designed around 16 kHz audio; if the browser returns a different rate, downstream audio processing may not handle it gracefully.
Consider adding a guard to detect mismatches early:
Suggested guard
const audioContext = new AudioContext({ sampleRate: 16000 })
audioContextRef.current = audioContext
console.log(`🎧 AudioContext created, sample rate: ${audioContext.sampleRate}Hz`)
+ if (audioContext.sampleRate !== 16000) {
+ console.warn(`⚠️ Requested 16kHz but got ${audioContext.sampleRate}Hz — verify backend can handle this rate`)
+ }🤖 Prompt for AI Agents
In `@backends/advanced/webui/src/contexts/RecordingContext.tsx` around lines 479 -
483, After creating AudioContext in RecordingContext (audioContextRef.current =
audioContext in the AudioContext({ sampleRate: 16000 }) block), check
audioContext.sampleRate and if it !== 16000, fail-fast or perform client-side
resampling before sending audio to the backend: log a clear warning, close the
created AudioContext if necessary, then either (a) create an OfflineAudioContext
or use an AudioWorklet/ScriptProcessor to resample to 16 kHz and set a flag to
mark streams as 16k, or (b) reject/start-stream with an explicit error so the
backend won’t receive mismatched-rate audio; ensure the audio-start message and
downstream processing use the actual output sample rate (16k) after resampling
and reference audioContextRef/currentAudioSampleRate and the code that sends the
audio-start WebSocket message to implement this guard.
| @@ -0,0 +1,21 @@ | |||
| MIT License | |||
|
|
|||
| Copyright (c) 2024 Chronicle AI Contributors | |||
There was a problem hiding this comment.
Copyright year may be outdated.
This is a new file being added in 2026, but the copyright year is 2024. Consider updating to 2025 or 2026 to reflect when this derivative work was actually created. The 2024 date in the NOTICE file correctly attributes the original OMI SDK authors.
🤖 Prompt for AI Agents
In `@extras/friend-lite-sdk/LICENSE` at line 3, Update the copyright year in the
LICENSE file from "2024" to the correct year for this derivative work (e.g.,
"2026"); locate the line containing "Copyright (c) 2024 Chronicle AI
Contributors" and replace the year so the LICENSE reflects the actual creation
year of this addition.
| import asyncio | ||
| from friend_lite import OmiConnection, ButtonState, parse_button_event | ||
|
|
||
| async def main(): | ||
| async with OmiConnection("AA:BB:CC:DD:EE:FF") as conn: | ||
| await conn.subscribe_audio(lambda _handle, data: print(len(data), "bytes")) | ||
| await conn.wait_until_disconnected() | ||
|
|
||
| asyncio.run(main()) |
There was a problem hiding this comment.
Unused imports in the usage example.
ButtonState and parse_button_event are imported but never used in the code snippet. Either trim the import to just OmiConnection, or extend the example to demonstrate button event handling.
🤖 Prompt for AI Agents
In `@extras/friend-lite-sdk/README.md` around lines 22 - 30, The example imports
ButtonState and parse_button_event but doesn't use them; either remove them from
the import and keep only OmiConnection, or update the example to demonstrate
button handling by calling the connection's button subscription (e.g., using a
subscribe_button or similar callback), passing events into parse_button_event
and checking ButtonState to show handling alongside the existing subscribe_audio
and wait_until_disconnected calls; ensure you reference OmiConnection,
parse_button_event, and ButtonState in the updated snippet so no imported
symbols are unused.
| try: | ||
| loop = asyncio.get_running_loop() | ||
| loop.create_task(send_button_event(state.name)) | ||
| except RuntimeError: | ||
| logger.debug("No running event loop, cannot send button event") |
There was a problem hiding this comment.
Fire-and-forget task may be garbage-collected, silently dropping button events.
loop.create_task(...) returns a Task that is immediately discarded. CPython's GC can collect it before completion, and any exception will be silently swallowed. Store a reference to keep the task alive.
🐛 Proposed fix
+# Module-level set to prevent GC of fire-and-forget tasks
+_background_tasks: set = set()
+
def handle_button_event(sender: Any, data: bytes) -> None:
try:
state = parse_button_event(data)
except Exception as e:
logger.error("Button event parse error: %s", e)
return
if state != ButtonState.IDLE:
logger.info("Button event: %s", state.name)
try:
loop = asyncio.get_running_loop()
- loop.create_task(send_button_event(state.name))
+ task = loop.create_task(send_button_event(state.name))
+ _background_tasks.add(task)
+ task.add_done_callback(_background_tasks.discard)
except RuntimeError:
logger.debug("No running event loop, cannot send button event")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| try: | |
| loop = asyncio.get_running_loop() | |
| loop.create_task(send_button_event(state.name)) | |
| except RuntimeError: | |
| logger.debug("No running event loop, cannot send button event") | |
| # Module-level set to prevent GC of fire-and-forget tasks | |
| _background_tasks: set = set() | |
| def handle_button_event(sender: Any, data: bytes) -> None: | |
| try: | |
| state = parse_button_event(data) | |
| except Exception as e: | |
| logger.error("Button event parse error: %s", e) | |
| return | |
| if state != ButtonState.IDLE: | |
| logger.info("Button event: %s", state.name) | |
| try: | |
| loop = asyncio.get_running_loop() | |
| task = loop.create_task(send_button_event(state.name)) | |
| _background_tasks.add(task) | |
| task.add_done_callback(_background_tasks.discard) | |
| except RuntimeError: | |
| logger.debug("No running event loop, cannot send button event") |
🧰 Tools
🪛 Ruff (0.14.14)
[warning] 76-76: Store a reference to the return value of loop.create_task
(RUF006)
🤖 Prompt for AI Agents
In `@extras/local-omi-bt/connect-omi.py` around lines 74 - 78, The created Task
from loop.create_task(send_button_event(state.name)) is immediately discarded
and may be GC'd; keep a strong reference and remove it when done. Replace the
discard with storing the Task in a module- or instance-level collection (e.g., a
set/list named _pending_tasks), create the Task via
loop.create_task(send_button_event(...)), add a done callback to remove it from
_pending_tasks and log exceptions if task.exception() is set; reference
send_button_event and loop.create_task when making these edits.
| scipy>=1.12.0 | ||
| opuslib>=3.0.1 | ||
| friend-lite-sdk | ||
| friend-lite-sdk @ file:../friend-lite-sdk |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify the friend-lite-sdk directory and check for documentation
echo "=== Checking for friend-lite-sdk directory ==="
if [ -d "extras/friend-lite-sdk" ]; then
echo "✓ Directory exists: extras/friend-lite-sdk"
ls -la extras/friend-lite-sdk/ | head -20
else
echo "✗ Directory NOT found: extras/friend-lite-sdk"
fi
echo -e "\n=== Checking for package configuration ==="
if [ -f "extras/friend-lite-sdk/setup.py" ]; then
echo "✓ Found setup.py"
fi
if [ -f "extras/friend-lite-sdk/pyproject.toml" ]; then
echo "✓ Found pyproject.toml"
head -20 extras/friend-lite-sdk/pyproject.toml
fi
echo -e "\n=== Checking for documentation about local setup ==="
find . -name "README*" -o -name "SETUP*" -o -name "DEVELOPMENT*" | grep -E "(README|SETUP|DEVELOPMENT)" | head -10Repository: SimpleOpenSoftware/chronicle
Length of output: 1561
🏁 Script executed:
#!/bin/bash
# Check for references to friend-lite-sdk in documentation and CI/CD
echo "=== Searching for documentation about local dependencies ==="
rg -l "friend-lite-sdk|local.*setup|development.*setup" --type md --type txt . 2>/dev/null | head -10
echo -e "\n=== Checking CI/CD configuration ==="
if [ -f ".github/workflows" ]; then
find .github/workflows -type f -name "*.yml" -o -name "*.yaml" | head -5
fi
ls -la .github/workflows/ 2>/dev/null | head -10Repository: SimpleOpenSoftware/chronicle
Length of output: 951
🏁 Script executed:
#!/bin/bash
# Check the actual requirements.txt and surrounding context
echo "=== Viewing requirements.txt content ==="
cat -n extras/local-omi-bt/requirements.txt
echo -e "\n=== Checking directory structure ==="
ls -la extras/Repository: SimpleOpenSoftware/chronicle
Length of output: 1128
🏁 Script executed:
#!/bin/bash
# Check documentation for local setup instructions
echo "=== Checking quickstart.md ==="
rg -A 5 -B 5 "friend-lite|local-omi|requirements" ./quickstart.md
echo -e "\n=== Checking friend-lite-sdk README ==="
cat extras/friend-lite-sdk/README.md
echo -e "\n=== Checking local-omi-bt context ==="
ls -la extras/local-omi-bt/Repository: SimpleOpenSoftware/chronicle
Length of output: 2930
🏁 Script executed:
#!/bin/bash
# Check if CI/CD workflows handle the local dependency
echo "=== Checking for mentions of local-omi-bt or friend-lite-sdk in CI/CD ==="
rg -l "local-omi-bt|friend-lite-sdk" .github/workflows/
echo -e "\n=== Check if setup.py exists for local-omi-bt ==="
if [ -f "extras/local-omi-bt/setup.py" ]; then
echo "✓ Found setup.py in local-omi-bt"
fi
if [ -f "extras/local-omi-bt/pyproject.toml" ]; then
echo "✓ Found pyproject.toml in local-omi-bt"
fi
ls -la extras/local-omi-bt/Repository: SimpleOpenSoftware/chronicle
Length of output: 692
🏁 Script executed:
#!/bin/bash
# Check main README and other setup docs
echo "=== Checking main README for local-omi-bt setup ==="
rg -A 10 -B 5 "local-omi-bt|OMI.*BLE|Friend.*Lite" ./README.md | head -50Repository: SimpleOpenSoftware/chronicle
Length of output: 123
Use editable install (-e) instead of file path reference, consistent with friend-lite-sdk's documentation.
The friend-lite-sdk/README.md explicitly recommends installing with pip install -e extras/friend-lite-sdk, but this requirements.txt uses @ file:../friend-lite-sdk instead. While the path exists and resolves correctly, the @ file: syntax does not provide editable install behavior and may not work consistently across all pip contexts (wheels, build operations, etc.).
Update to use the editable install approach documented in the package:
-e ../friend-lite-sdk
Or if editable install is not preferred, add documentation explaining why the file reference is used instead of the recommended approach.
🤖 Prompt for AI Agents
In `@extras/local-omi-bt/requirements.txt` at line 5, Replace the non-editable
file URL dependency "friend-lite-sdk @ file:../friend-lite-sdk" in the
requirements entry with an editable install as recommended by the SDK docs: use
"-e ../friend-lite-sdk" to enable development/editable installs for
friend-lite-sdk; if you intentionally prefer a non-editable install, add a short
comment in the requirements (or README) explaining why the file: reference was
chosen instead of the editable "-e" form.
| # Look for neo/neosapien devices | ||
| if d.name and any(x in d.name.lower() for x in ["neo", "sapien"]): | ||
| neo_devices.append(d) |
There was a problem hiding this comment.
Device name filter is inconsistent with connect-omi.py.
This script filters for "neo" and "sapien", but connect-omi.py (line ~89 in the relevant snippet) filters for "omi" and "neo". This means:
scan_devices.pywon't find devices named "OMI-xxx" (no "sapien" or "neo" in name)connect-omi.pywon't find devices named "neosapien-xxx" without "neo" already matching
Consider aligning the filters so both scripts discover the same set of devices.
Suggested fix
- if d.name and any(x in d.name.lower() for x in ["neo", "sapien"]):
+ if d.name and any(x in d.name.lower() for x in ["neo", "sapien", "omi"]):📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # Look for neo/neosapien devices | |
| if d.name and any(x in d.name.lower() for x in ["neo", "sapien"]): | |
| neo_devices.append(d) | |
| # Look for neo/neosapien devices | |
| if d.name and any(x in d.name.lower() for x in ["neo", "sapien", "omi"]): | |
| neo_devices.append(d) |
🤖 Prompt for AI Agents
In `@extras/local-omi-bt/scan_devices.py` around lines 17 - 19, The device name
filter in scan_devices.py (the any(...) check that appends to neo_devices based
on d.name) is inconsistent with connect-omi.py; change the predicate used in the
any(...) call so both scripts use the same token set (e.g., include "omi" and
"neo" rather than "neo" and "sapien") so that devices like "OMI-xxx" and
"neosapien-xxx" are discovered consistently; update the any(...) token list
inside the d.name check where neo_devices.append(d) is called (or centralize the
token list into a shared constant if preferred) to match connect-omi.py’s
filter.
- Increased the maximum event log size in PluginRouter from 200 to 1000 for improved event tracking. - Refactored audio stream producer to dynamically read audio format from Redis session metadata, enhancing flexibility in audio handling. - Updated transcription job processing to utilize session-specific audio format settings, ensuring accurate audio processing. - Enhanced audio file writing utility to accept PCM parameters, allowing for better control over audio data handling.
- Introduced a new `markers` list in `ClientState` to track button event data during conversations. - Updated comment in `open_conversation_job` to clarify the behavior of the `timeout_triggered` variable, ensuring better understanding of session management.
- Updated audio processing logs to consistently use the `filename` variable instead of `file.filename` for clarity. - Enhanced error logging to utilize the `filename` variable, improving traceability of issues during audio processing. - Adjusted title generation logic to handle cases where the filename is "unknown," ensuring a default title is used. - Minor refactor in conversation closing logs to use `user.user_id` for better consistency in user identification.
|
| Metric | Count |
|---|---|
| ✅ Passed | 102 |
| ❌ Failed | 20 |
| 📊 Total | 122 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html-no-api - HTML reports
- robot-test-results-xml-no-api - XML output
|
| Metric | Count |
|---|---|
| ✅ Passed | 102 |
| ❌ Failed | 20 |
| 📊 Total | 122 |
📊 View Reports
GitHub Pages (Live Reports):
Download Artifacts:
- robot-test-reports-html-no-api - HTML reports
- robot-test-results-xml-no-api - XML output
- Updated `get_conversations` function to support pagination through `limit` and `offset` parameters, improving performance for large datasets. - Consolidated query logic to fetch both normal and orphan conversations in a single database call, reducing round-trips and enhancing efficiency. - Modified the response structure to include total count, limit, and offset in the returned data for better client-side handling. - Adjusted database indexing to optimize queries for paginated results, ensuring faster access to conversation data.
- Moved connection logging for the Wyoming server to a more structured format within the `transcribe_wyoming` function. - Ensured that connection attempts and successes are logged consistently for better traceability during audio transcription processes.
AnkushMalaker
merged commit 79085ff
into
feat/better-reprocess-memory
* Enhance ASR service descriptions and provider feedback in wizard.py (#290) - Updated the description for the 'asr-services' to remove the specific mention of 'Parakeet', making it more general. - Improved the console output for auto-selected services to include the transcription provider label, enhancing user feedback during service selection. * Refactor Obsidian and Knowledge Graph integration in services and setup - Removed redundant Obsidian and Knowledge Graph configuration checks from services.py, streamlining the command execution process. - Updated wizard.py to enhance user experience by setting default options for speaker recognition during service selection. - Improved Neo4j password handling in setup processes, ensuring consistent configuration prompts and feedback. - Introduced a new cron scheduler for managing scheduled tasks, enhancing the backend's automation capabilities. - Added new entity annotation features, allowing for corrections and updates to knowledge graph entities directly through the API. * Enhance ASR services configuration and VibeVoice integration - Added new configuration options for VibeVoice ASR in defaults.yml, including batching parameters for audio processing. - Updated Docker Compose files to mount the config directory, ensuring access to ASR service configurations. - Enhanced the VibeVoice transcriber to load configuration settings from defaults.yml, allowing for dynamic adjustments via environment variables. - Introduced quantization options for model loading in the VibeVoice transcriber, improving performance and flexibility. - Refactored the speaker identification process to streamline audio handling and improve logging for better debugging. - Updated documentation to reflect new configuration capabilities and usage instructions for the VibeVoice ASR provider. * Enhance LangFuse integration and memory reprocessing capabilities - Introduced functions for checking LangFuse configuration in services.py, ensuring proper setup for observability. - Updated wizard.py to facilitate user input for LangFuse configuration, including options for local and external setups. - Implemented memory reprocessing logic in memory services to update existing memories based on speaker re-identification. - Enhanced speaker recognition client to support per-segment identification, improving accuracy during reprocessing. - Refactored various components to streamline handling of LangFuse parameters and improve overall service management. * Enhance service management and user input handling - Updated services.py to include LangFuse configuration checks during service startup, improving observability setup. - Refactored wizard.py to utilize a masked input for Neo4j password prompts, enhancing user experience and security. - Improved cron scheduler in advanced_omi_backend to manage active tasks and validate cron expressions, ensuring robust job execution. - Enhanced speaker recognition client documentation to clarify user_id limitations, preparing for future multi-user support. - Updated knowledge graph routes to enforce validation on entity updates, ensuring at least one field is provided for updates. * fix: Plugin System Refactor (#301) * Refactor connect-omi.py for improved device selection and user interaction - Replaced references to the chronicle Bluetooth library with friend_lite for device management. - Removed the list_devices function and implemented a new prompt_user_to_pick_device function to enhance user interaction when selecting OMI/Neo devices. - Updated the find_and_set_omi_mac function to utilize the new device selection method, improving the overall flow of device connection. - Added a new scan_devices.py script for quick scanning of neo/neosapien devices, enhancing usability. - Updated README.md to reflect new usage instructions and prerequisites for connecting to OMI devices over Bluetooth. - Enhanced start.sh to ensure proper environment variable setup for macOS users. * Add friend-lite-sdk: Initial implementation of Python SDK for OMI/Friend Lite BLE devices - Introduced the friend-lite-sdk, a Python SDK for OMI/Friend Lite BLE devices, enabling audio streaming, button events, and transcription functionalities. - Added LICENSE and NOTICE files to clarify licensing and attribution. - Created pyproject.toml for package management, specifying dependencies and project metadata. - Developed core modules including bluetooth connection handling, button event parsing, audio decoding, and transcription capabilities. - Implemented example usage in README.md to guide users on installation and basic functionality. - Enhanced connect-omi.py to utilize the new SDK for improved device management and event handling. - Updated requirements.txt to reference the new SDK for local development. This commit lays the foundation for further enhancements and integrations with OMI devices. * Enhance client state and plugin architecture for button event handling - Introduced a new `markers` list in `ClientState` to collect button event data during sessions. - Added `add_marker` method to facilitate the addition of markers to the current session. - Implemented `on_button_event` method in the `BasePlugin` class to handle device button events, providing context data for button state and timestamps. - Updated `PluginRouter` to route button events to the appropriate plugin handler. - Enhanced conversation job handling to attach markers from Redis sessions, improving the tracking of button events during conversations. * Move plugins locatino - Introduced the Email Summarizer plugin that automatically sends email summaries upon conversation completion. - Implemented SMTP email service for sending formatted HTML and plain text emails. - Added configuration options for SMTP settings and email content in `config.yml`. - Created setup script for easy configuration of SMTP credentials and plugin orchestration. - Enhanced documentation with usage instructions and troubleshooting tips for the plugin. - Updated existing plugin architecture to support new event handling for email summaries. * Enhance Docker Compose and Plugin Management - Added external plugins directory to Docker Compose files for better plugin management. - Updated environment variables for MongoDB and Redis services to ensure consistent behavior. - Introduced new dependencies in `uv.lock` for improved functionality. - Refactored audio processing to support various audio formats and enhance error handling. - Implemented new plugin event types and services for better integration and communication between plugins. - Enhanced conversation and session management to support new closing mechanisms and event logging. * Update audio processing and event logging - Increased the maximum event log size in PluginRouter from 200 to 1000 for improved event tracking. - Refactored audio stream producer to dynamically read audio format from Redis session metadata, enhancing flexibility in audio handling. - Updated transcription job processing to utilize session-specific audio format settings, ensuring accurate audio processing. - Enhanced audio file writing utility to accept PCM parameters, allowing for better control over audio data handling. * Add markers list to ClientState and update timeout trigger comment - Introduced a new `markers` list in `ClientState` to track button event data during conversations. - Updated comment in `open_conversation_job` to clarify the behavior of the `timeout_triggered` variable, ensuring better understanding of session management. * Refactor audio file logging and error handling - Updated audio processing logs to consistently use the `filename` variable instead of `file.filename` for clarity. - Enhanced error logging to utilize the `filename` variable, improving traceability of issues during audio processing. - Adjusted title generation logic to handle cases where the filename is "unknown," ensuring a default title is used. - Minor refactor in conversation closing logs to use `user.user_id` for better consistency in user identification. * Enhance conversation retrieval with pagination and orphan handling - Updated `get_conversations` function to support pagination through `limit` and `offset` parameters, improving performance for large datasets. - Consolidated query logic to fetch both normal and orphan conversations in a single database call, reducing round-trips and enhancing efficiency. - Modified the response structure to include total count, limit, and offset in the returned data for better client-side handling. - Adjusted database indexing to optimize queries for paginated results, ensuring faster access to conversation data. * Refactor connection logging in transcribe function - Moved connection logging for the Wyoming server to a more structured format within the `transcribe_wyoming` function. - Ensured that connection attempts and successes are logged consistently for better traceability during audio transcription processes.
* Enhance ASR service descriptions and provider feedback in wizard.py - Updated the description for the 'asr-services' to remove the specific mention of 'Parakeet', making it more general. - Improved the console output for auto-selected services to include the transcription provider label, enhancing user feedback during service selection. * Implement LangFuse integration for observability and prompt management - Added LangFuse configuration options in the .env.template for observability and prompt management. - Introduced setup_langfuse method in ChronicleSetup to handle LangFuse initialization and configuration prompts. - Enhanced prompt management by integrating a centralized PromptRegistry for dynamic prompt retrieval and registration. - Updated various services to utilize prompts from the PromptRegistry, improving flexibility and maintainability. - Refactored OpenAI client initialization to support optional LangFuse tracing, enhancing observability during API interactions. - Added new prompt defaults for memory management and conversation handling, ensuring consistent behavior across the application. * Enhance LangFuse integration and service management - Added LangFuse service configuration in services.py and wizard.py, including paths, commands, and descriptions. - Implemented auto-selection for LangFuse during service setup, improving user experience. - Enhanced service startup process to display prompt management tips for LangFuse, guiding users on editing AI prompts. - Updated run_service_setup to handle LangFuse-specific parameters, including admin credentials and API keys, ensuring seamless integration with backend services. * Feat/better reprocess memory (#300) * Enhance ASR service descriptions and provider feedback in wizard.py (#290) - Updated the description for the 'asr-services' to remove the specific mention of 'Parakeet', making it more general. - Improved the console output for auto-selected services to include the transcription provider label, enhancing user feedback during service selection. * Refactor Obsidian and Knowledge Graph integration in services and setup - Removed redundant Obsidian and Knowledge Graph configuration checks from services.py, streamlining the command execution process. - Updated wizard.py to enhance user experience by setting default options for speaker recognition during service selection. - Improved Neo4j password handling in setup processes, ensuring consistent configuration prompts and feedback. - Introduced a new cron scheduler for managing scheduled tasks, enhancing the backend's automation capabilities. - Added new entity annotation features, allowing for corrections and updates to knowledge graph entities directly through the API. * Enhance ASR services configuration and VibeVoice integration - Added new configuration options for VibeVoice ASR in defaults.yml, including batching parameters for audio processing. - Updated Docker Compose files to mount the config directory, ensuring access to ASR service configurations. - Enhanced the VibeVoice transcriber to load configuration settings from defaults.yml, allowing for dynamic adjustments via environment variables. - Introduced quantization options for model loading in the VibeVoice transcriber, improving performance and flexibility. - Refactored the speaker identification process to streamline audio handling and improve logging for better debugging. - Updated documentation to reflect new configuration capabilities and usage instructions for the VibeVoice ASR provider. * Enhance LangFuse integration and memory reprocessing capabilities - Introduced functions for checking LangFuse configuration in services.py, ensuring proper setup for observability. - Updated wizard.py to facilitate user input for LangFuse configuration, including options for local and external setups. - Implemented memory reprocessing logic in memory services to update existing memories based on speaker re-identification. - Enhanced speaker recognition client to support per-segment identification, improving accuracy during reprocessing. - Refactored various components to streamline handling of LangFuse parameters and improve overall service management. * Enhance service management and user input handling - Updated services.py to include LangFuse configuration checks during service startup, improving observability setup. - Refactored wizard.py to utilize a masked input for Neo4j password prompts, enhancing user experience and security. - Improved cron scheduler in advanced_omi_backend to manage active tasks and validate cron expressions, ensuring robust job execution. - Enhanced speaker recognition client documentation to clarify user_id limitations, preparing for future multi-user support. - Updated knowledge graph routes to enforce validation on entity updates, ensuring at least one field is provided for updates. * fix: Plugin System Refactor (#301) * Refactor connect-omi.py for improved device selection and user interaction - Replaced references to the chronicle Bluetooth library with friend_lite for device management. - Removed the list_devices function and implemented a new prompt_user_to_pick_device function to enhance user interaction when selecting OMI/Neo devices. - Updated the find_and_set_omi_mac function to utilize the new device selection method, improving the overall flow of device connection. - Added a new scan_devices.py script for quick scanning of neo/neosapien devices, enhancing usability. - Updated README.md to reflect new usage instructions and prerequisites for connecting to OMI devices over Bluetooth. - Enhanced start.sh to ensure proper environment variable setup for macOS users. * Add friend-lite-sdk: Initial implementation of Python SDK for OMI/Friend Lite BLE devices - Introduced the friend-lite-sdk, a Python SDK for OMI/Friend Lite BLE devices, enabling audio streaming, button events, and transcription functionalities. - Added LICENSE and NOTICE files to clarify licensing and attribution. - Created pyproject.toml for package management, specifying dependencies and project metadata. - Developed core modules including bluetooth connection handling, button event parsing, audio decoding, and transcription capabilities. - Implemented example usage in README.md to guide users on installation and basic functionality. - Enhanced connect-omi.py to utilize the new SDK for improved device management and event handling. - Updated requirements.txt to reference the new SDK for local development. This commit lays the foundation for further enhancements and integrations with OMI devices. * Enhance client state and plugin architecture for button event handling - Introduced a new `markers` list in `ClientState` to collect button event data during sessions. - Added `add_marker` method to facilitate the addition of markers to the current session. - Implemented `on_button_event` method in the `BasePlugin` class to handle device button events, providing context data for button state and timestamps. - Updated `PluginRouter` to route button events to the appropriate plugin handler. - Enhanced conversation job handling to attach markers from Redis sessions, improving the tracking of button events during conversations. * Move plugins locatino - Introduced the Email Summarizer plugin that automatically sends email summaries upon conversation completion. - Implemented SMTP email service for sending formatted HTML and plain text emails. - Added configuration options for SMTP settings and email content in `config.yml`. - Created setup script for easy configuration of SMTP credentials and plugin orchestration. - Enhanced documentation with usage instructions and troubleshooting tips for the plugin. - Updated existing plugin architecture to support new event handling for email summaries. * Enhance Docker Compose and Plugin Management - Added external plugins directory to Docker Compose files for better plugin management. - Updated environment variables for MongoDB and Redis services to ensure consistent behavior. - Introduced new dependencies in `uv.lock` for improved functionality. - Refactored audio processing to support various audio formats and enhance error handling. - Implemented new plugin event types and services for better integration and communication between plugins. - Enhanced conversation and session management to support new closing mechanisms and event logging. * Update audio processing and event logging - Increased the maximum event log size in PluginRouter from 200 to 1000 for improved event tracking. - Refactored audio stream producer to dynamically read audio format from Redis session metadata, enhancing flexibility in audio handling. - Updated transcription job processing to utilize session-specific audio format settings, ensuring accurate audio processing. - Enhanced audio file writing utility to accept PCM parameters, allowing for better control over audio data handling. * Add markers list to ClientState and update timeout trigger comment - Introduced a new `markers` list in `ClientState` to track button event data during conversations. - Updated comment in `open_conversation_job` to clarify the behavior of the `timeout_triggered` variable, ensuring better understanding of session management. * Refactor audio file logging and error handling - Updated audio processing logs to consistently use the `filename` variable instead of `file.filename` for clarity. - Enhanced error logging to utilize the `filename` variable, improving traceability of issues during audio processing. - Adjusted title generation logic to handle cases where the filename is "unknown," ensuring a default title is used. - Minor refactor in conversation closing logs to use `user.user_id` for better consistency in user identification. * Enhance conversation retrieval with pagination and orphan handling - Updated `get_conversations` function to support pagination through `limit` and `offset` parameters, improving performance for large datasets. - Consolidated query logic to fetch both normal and orphan conversations in a single database call, reducing round-trips and enhancing efficiency. - Modified the response structure to include total count, limit, and offset in the returned data for better client-side handling. - Adjusted database indexing to optimize queries for paginated results, ensuring faster access to conversation data. * Refactor connection logging in transcribe function - Moved connection logging for the Wyoming server to a more structured format within the `transcribe_wyoming` function. - Ensured that connection attempts and successes are logged consistently for better traceability during audio transcription processes. * Feat/neo sdk (#302) * Update friend-lite-sdk for Neo1 device support and enhance documentation - Updated the friend-lite-sdk to version 0.3.0, reflecting the transition to support OMI/Neo1 BLE wearable devices. - Refactored the Bluetooth connection handling to introduce a new `WearableConnection` class, enhancing the connection lifecycle management for wearable devices. - Added a new `Neo1Connection` class for controlling Neo1 devices, including methods for sleep and wake functionalities. - Updated UUID constants to include Neo1-specific characteristics, improving device interaction capabilities. - Revised the plugin development guide to reflect changes in device naming and connection processes. - Removed outdated local OMI Bluetooth scripts and documentation to streamline the project structure and focus on wearable client development. * Refactor backend audio streaming to use Opus codec and enhance menu app functionality - Updated backend_sender.py to stream raw Opus audio instead of PCM, improving bandwidth efficiency. - Modified stream_to_backend function to handle Opus audio data and adjusted audio chunk parameters accordingly. - Enhanced main.py with new CLI commands for device scanning and connection management, improving user experience. - Introduced menu_app.py for a macOS menu bar application, providing a user-friendly interface for device management and status display. - Added README.md to document usage instructions and configuration details for the local wearable client. - Updated requirements.txt to include new dependencies for the menu app and service management. - Implemented service.py for managing launchd service installation and configuration on macOS, enabling auto-start on login. * Refactor audio processing and queue management in local wearable client - Removed the audio queue in favor of a dedicated BLE data queue and backend queue for improved data handling. - Enhanced the `connect_and_stream` function to streamline audio decoding and writing to the local file sink. - Updated the handling of BLE data to ensure robust queue management and error logging. - Improved task management during device disconnection to ensure proper cleanup and error handling. - Updated requirements.txt to specify a minimum version for easy_audio_interfaces, ensuring compatibility.
* Enhance ASR service descriptions and provider feedback in wizard.py - Updated the description for the 'asr-services' to remove the specific mention of 'Parakeet', making it more general. - Improved the console output for auto-selected services to include the transcription provider label, enhancing user feedback during service selection. * Implement LangFuse integration for observability and prompt management - Added LangFuse configuration options in the .env.template for observability and prompt management. - Introduced setup_langfuse method in ChronicleSetup to handle LangFuse initialization and configuration prompts. - Enhanced prompt management by integrating a centralized PromptRegistry for dynamic prompt retrieval and registration. - Updated various services to utilize prompts from the PromptRegistry, improving flexibility and maintainability. - Refactored OpenAI client initialization to support optional LangFuse tracing, enhancing observability during API interactions. - Added new prompt defaults for memory management and conversation handling, ensuring consistent behavior across the application. * Enhance LangFuse integration and service management - Added LangFuse service configuration in services.py and wizard.py, including paths, commands, and descriptions. - Implemented auto-selection for LangFuse during service setup, improving user experience. - Enhanced service startup process to display prompt management tips for LangFuse, guiding users on editing AI prompts. - Updated run_service_setup to handle LangFuse-specific parameters, including admin credentials and API keys, ensuring seamless integration with backend services. * Feat/better reprocess memory (#300) * Enhance ASR service descriptions and provider feedback in wizard.py (#290) - Updated the description for the 'asr-services' to remove the specific mention of 'Parakeet', making it more general. - Improved the console output for auto-selected services to include the transcription provider label, enhancing user feedback during service selection. * Refactor Obsidian and Knowledge Graph integration in services and setup - Removed redundant Obsidian and Knowledge Graph configuration checks from services.py, streamlining the command execution process. - Updated wizard.py to enhance user experience by setting default options for speaker recognition during service selection. - Improved Neo4j password handling in setup processes, ensuring consistent configuration prompts and feedback. - Introduced a new cron scheduler for managing scheduled tasks, enhancing the backend's automation capabilities. - Added new entity annotation features, allowing for corrections and updates to knowledge graph entities directly through the API. * Enhance ASR services configuration and VibeVoice integration - Added new configuration options for VibeVoice ASR in defaults.yml, including batching parameters for audio processing. - Updated Docker Compose files to mount the config directory, ensuring access to ASR service configurations. - Enhanced the VibeVoice transcriber to load configuration settings from defaults.yml, allowing for dynamic adjustments via environment variables. - Introduced quantization options for model loading in the VibeVoice transcriber, improving performance and flexibility. - Refactored the speaker identification process to streamline audio handling and improve logging for better debugging. - Updated documentation to reflect new configuration capabilities and usage instructions for the VibeVoice ASR provider. * Enhance LangFuse integration and memory reprocessing capabilities - Introduced functions for checking LangFuse configuration in services.py, ensuring proper setup for observability. - Updated wizard.py to facilitate user input for LangFuse configuration, including options for local and external setups. - Implemented memory reprocessing logic in memory services to update existing memories based on speaker re-identification. - Enhanced speaker recognition client to support per-segment identification, improving accuracy during reprocessing. - Refactored various components to streamline handling of LangFuse parameters and improve overall service management. * Enhance service management and user input handling - Updated services.py to include LangFuse configuration checks during service startup, improving observability setup. - Refactored wizard.py to utilize a masked input for Neo4j password prompts, enhancing user experience and security. - Improved cron scheduler in advanced_omi_backend to manage active tasks and validate cron expressions, ensuring robust job execution. - Enhanced speaker recognition client documentation to clarify user_id limitations, preparing for future multi-user support. - Updated knowledge graph routes to enforce validation on entity updates, ensuring at least one field is provided for updates. * fix: Plugin System Refactor (#301) * Refactor connect-omi.py for improved device selection and user interaction - Replaced references to the chronicle Bluetooth library with friend_lite for device management. - Removed the list_devices function and implemented a new prompt_user_to_pick_device function to enhance user interaction when selecting OMI/Neo devices. - Updated the find_and_set_omi_mac function to utilize the new device selection method, improving the overall flow of device connection. - Added a new scan_devices.py script for quick scanning of neo/neosapien devices, enhancing usability. - Updated README.md to reflect new usage instructions and prerequisites for connecting to OMI devices over Bluetooth. - Enhanced start.sh to ensure proper environment variable setup for macOS users. * Add friend-lite-sdk: Initial implementation of Python SDK for OMI/Friend Lite BLE devices - Introduced the friend-lite-sdk, a Python SDK for OMI/Friend Lite BLE devices, enabling audio streaming, button events, and transcription functionalities. - Added LICENSE and NOTICE files to clarify licensing and attribution. - Created pyproject.toml for package management, specifying dependencies and project metadata. - Developed core modules including bluetooth connection handling, button event parsing, audio decoding, and transcription capabilities. - Implemented example usage in README.md to guide users on installation and basic functionality. - Enhanced connect-omi.py to utilize the new SDK for improved device management and event handling. - Updated requirements.txt to reference the new SDK for local development. This commit lays the foundation for further enhancements and integrations with OMI devices. * Enhance client state and plugin architecture for button event handling - Introduced a new `markers` list in `ClientState` to collect button event data during sessions. - Added `add_marker` method to facilitate the addition of markers to the current session. - Implemented `on_button_event` method in the `BasePlugin` class to handle device button events, providing context data for button state and timestamps. - Updated `PluginRouter` to route button events to the appropriate plugin handler. - Enhanced conversation job handling to attach markers from Redis sessions, improving the tracking of button events during conversations. * Move plugins locatino - Introduced the Email Summarizer plugin that automatically sends email summaries upon conversation completion. - Implemented SMTP email service for sending formatted HTML and plain text emails. - Added configuration options for SMTP settings and email content in `config.yml`. - Created setup script for easy configuration of SMTP credentials and plugin orchestration. - Enhanced documentation with usage instructions and troubleshooting tips for the plugin. - Updated existing plugin architecture to support new event handling for email summaries. * Enhance Docker Compose and Plugin Management - Added external plugins directory to Docker Compose files for better plugin management. - Updated environment variables for MongoDB and Redis services to ensure consistent behavior. - Introduced new dependencies in `uv.lock` for improved functionality. - Refactored audio processing to support various audio formats and enhance error handling. - Implemented new plugin event types and services for better integration and communication between plugins. - Enhanced conversation and session management to support new closing mechanisms and event logging. * Update audio processing and event logging - Increased the maximum event log size in PluginRouter from 200 to 1000 for improved event tracking. - Refactored audio stream producer to dynamically read audio format from Redis session metadata, enhancing flexibility in audio handling. - Updated transcription job processing to utilize session-specific audio format settings, ensuring accurate audio processing. - Enhanced audio file writing utility to accept PCM parameters, allowing for better control over audio data handling. * Add markers list to ClientState and update timeout trigger comment - Introduced a new `markers` list in `ClientState` to track button event data during conversations. - Updated comment in `open_conversation_job` to clarify the behavior of the `timeout_triggered` variable, ensuring better understanding of session management. * Refactor audio file logging and error handling - Updated audio processing logs to consistently use the `filename` variable instead of `file.filename` for clarity. - Enhanced error logging to utilize the `filename` variable, improving traceability of issues during audio processing. - Adjusted title generation logic to handle cases where the filename is "unknown," ensuring a default title is used. - Minor refactor in conversation closing logs to use `user.user_id` for better consistency in user identification. * Enhance conversation retrieval with pagination and orphan handling - Updated `get_conversations` function to support pagination through `limit` and `offset` parameters, improving performance for large datasets. - Consolidated query logic to fetch both normal and orphan conversations in a single database call, reducing round-trips and enhancing efficiency. - Modified the response structure to include total count, limit, and offset in the returned data for better client-side handling. - Adjusted database indexing to optimize queries for paginated results, ensuring faster access to conversation data. * Refactor connection logging in transcribe function - Moved connection logging for the Wyoming server to a more structured format within the `transcribe_wyoming` function. - Ensured that connection attempts and successes are logged consistently for better traceability during audio transcription processes. * Refactor configuration management and enhance plugin architecture - Replaced PyYAML with ruamel.yaml for improved YAML handling, preserving quotes and enhancing configuration loading. - Updated ConfigManager to utilize ruamel.yaml for loading and saving configuration files, ensuring better error handling and validation. - Enhanced service startup messages to display access URLs for backend services, improving user experience. - Introduced new plugin health tracking in PluginRouter, allowing for better monitoring of plugin initialization and error states. - Refactored audio stream client and conversation management to streamline audio processing and improve error handling. - Updated Docker and requirements configurations to include ruamel.yaml, ensuring compatibility across environments. * refactor clean up script * cleanup partial mycelia integration * Refactor configuration management and remove Mycelia integration - Updated ConfigManager to remove references to the Mycelia memory provider, simplifying the memory provider options to only include "chronicle" and "openmemory_mcp". - Cleaned up Makefile by removing Mycelia-related targets and help descriptions, streamlining the build process. - Enhanced cleanup script documentation for clarity on usage and options. - Introduced LLM operation configurations to improve model management and prompt optimization capabilities. * Refactor Docker and cleanup scripts to remove 'uv' command usage - Updated cleanup.sh to directly execute the Python script without 'uv' command. - Modified Docker Compose files to remove 'uv run' from service commands, simplifying execution. - Enhanced start.sh to reflect changes in command usage and improve clarity in usage instructions. - Introduced a new transcription job timeout configuration in the backend, allowing for dynamic timeout settings. - Added insert annotation functionality in the API, enabling users to insert new segments in conversations. - Implemented memory retrieval for conversations, enhancing the ability to fetch related memories. - Improved error handling and logging across various modules for better traceability and debugging. * Add backend worker health check and job clearing functionality - Introduced a new function `get_backend_worker_health` to retrieve health metrics from the backend's /health endpoint, including worker count and queue status. - Updated `show_quick_status` to display worker health information, alerting users to potential issues with registered workers. - Added a new API endpoint `/jobs` to allow admin users to clear finished and failed jobs from all queues, enhancing job management capabilities. - Updated the frontend Queue component to include a button for clearing jobs, improving user interaction and management of job statuses. * Update plugin event descriptions and refactor event handling - Reduced redundancy by embedding descriptions directly within the PluginEvent enum, enhancing clarity and maintainability. - Removed the EVENT_DESCRIPTIONS dictionary, streamlining the event handling process in the plugin assistant. - Updated references in the plugin assistant to utilize the new description attributes, ensuring consistent event metadata usage.
1 participant
Summary by CodeRabbit
New Features
Bug Fixes
Documentation