feat: Add Voice Mode API with WebSocket support and VAD processing for real-time audio interaction #6826
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔧 (frontend): Refactor import path for VoiceAssistant component 🔧 (frontend): Refactor class name for button in upload-file-button component 🔧 (frontend): Refactor class name for button in voice-button component 🔧 (frontend): Refactor class name for button in applies.css 🔧 (frontend): Refactor class name for button in styleUtils.ts
…s to enter their OpenAI API key for voice transcription ♻️ (index.tsx): Refactor VoiceAssistant component to check for the presence of OpenAI API key before starting voice transcription and show ApiKeyPopup component if key is missing 🔧 (apiKeyModal/index.tsx): Remove the obsolete APIKeyModal component as it is no longer needed after implementing ApiKeyPopup in the voice assistant feature
… OpenAI. Add components and hooks for handling audio recording, processing, and WebSocket communication. Implement functionality to start, stop recording, play audio chunks, handle WebSocket messages, and initialize audio context. Add support for entering API key for OpenAI.
…t component for recording and processing audio input 🔧 (use-post-voice.tsx): Remove unused file use-post-voice.tsx from the project ♻️ (use-handle-websocket-message.ts): Refactor useHandleWebsocketMessage function to improve readability and remove unnecessary console logs ♻️ (use-initialize-audio.ts): Refactor useInitializeAudio function to handle audio context creation and resume more efficiently ✅ (use-interrupt-playback.ts): Add useInterruptPlayback function to handle interrupting audio playback ✅ (use-start-conversation.ts): Add useStartConversation function to initiate a conversation using a WebSocket connection 📝 (chat-input.tsx): Update import path for VoiceAssistant component to match the new file structure
github-actions
bot
added
enhancement
CodSpeed Performance ReportMerging #6826 will degrade performances by 12.08%Comparing Summary
Benchmarks breakdown
|
… clear audio queue, stop playback, and send stop message to audio processor if it exists
github-actions
bot
added
enhancement
…from modifying the file contents
github-actions
bot
added
enhancement
github-actions
bot
added
enhancement
|
Dupe of #4642 ? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces several new features and improvements to the
langflowproject, including the addition of new dependencies, the implementation of a newvoice_modeAPI, and various enhancements to the MCP module. Below are the most important changes grouped by theme:New Dependencies
webrtcvadandscipyto the project dependencies inpyproject.toml.API Enhancements
voice_mode_routerandmcp_routerto the main API router insrc/backend/base/langflow/api/router.py. [1] [2]voice_mode_routerto the API v1 initialization insrc/backend/base/langflow/api/v1/__init__.py. [1] [2]MCP Module Improvements
enable_progress_notificationsvariable and updated thehandle_call_toolfunction to initialize it if not already set insrc/backend/base/langflow/api/v1/mcp.py. [1] [2]Voice Mode API Implementation
voice_modeAPI insrc/backend/base/langflow/api/v1/voice_mode.py, including WebSocket handling, VAD processing, and integration with OpenAI Realtime API.Utility Enhancements
create_tool_coroutinefunction and added a newcreate_input_schema_from_json_schemafunction insrc/backend/base/langflow/base/mcp/util.py. [1] [2]connect_to_servermethod to include a timeout mechanism insrc/backend/base/langflow/components/tools/mcp_sse.py. [1] [2]