diff --git a/AUDIO_PROVIDER_SUMMARY.md b/AUDIO_PROVIDER_SUMMARY.md new file mode 100644 index 00000000..81ec4a7a --- /dev/null +++ b/AUDIO_PROVIDER_SUMMARY.md @@ -0,0 +1,137 @@ +# Audio Provider System - Corrected Architecture + +## Summary + +Audio is now a proper provider capability system with **two separate capabilities**: + +1. **`audio_input`** - Audio SOURCES (mobile, Omi, desktop, file, UNode) +2. **`audio_consumer`** - Audio DESTINATIONS (Chronicle, Mycelia, relay, webhooks) + +## Architecture + +``` +┌─────────────────────┐ ┌──────────────────────┐ +│ Audio INPUT │ │ Audio CONSUMER │ +│ (Source/Provider) │ ────────> │ (Destination) │ +└─────────────────────┘ └──────────────────────┘ + + • Mobile App Mic • Chronicle + • Omi Device • Mycelia + • Desktop Mic • Multi-Destination + • Audio File Upload • Custom WebSocket + • UNode Device • Webhook +``` + +## Files Created/Modified + +### Configuration Files +- ✅ `config/capabilities.yaml` - Added `audio_input` and `audio_consumer` capabilities +- ✅ `config/providers/audio_input.yaml` - 5 input providers (mobile, omi, desktop, file, unode) +- ✅ `config/providers/audio_consumer.yaml` - 5 consumer providers (chronicle, mycelia, multi-dest, custom, webhook) +- ✅ `config/config.defaults.yaml` - Default selections + +### Backend API +- ✅ `ushadow/backend/src/routers/audio_provider.py` - Audio consumer API + - `GET /api/providers/audio_consumer/active` - Get where to send audio + - `GET /api/providers/audio_consumer/available` - List consumers + - `PUT /api/providers/audio_consumer/active` - Switch consumer +- ✅ `ushadow/backend/src/routers/audio_relay.py` - Multi-destination relay + - `WS /ws/audio/relay` - Fanout to multiple consumers +- ✅ `ushadow/backend/main.py` - Registered routers + +### Mobile App Integration +- ✅ `ushadow/mobile/app/services/audioProviderApi.ts` - Consumer discovery API +- ✅ `ushadow/mobile/app/hooks/useMultiDestinationStreamer.ts` - Multi-cast support + +### Documentation +- ✅ `docs/AUDIO_PROVIDER_ARCHITECTURE.md` - Complete architecture guide +- ✅ `MULTI_DESTINATION_AUDIO_EXAMPLE.md` - Relay examples + +## How It Works + +### Mobile App (Audio Input Provider) + +```typescript +// 1. Mobile app asks: "Where should I send my audio?" +const consumer = await getActiveAudioConsumer(baseUrl, token); +// Returns: { provider_id: "chronicle", websocket_url: "ws://chronicle:5001/...", ...} + +// 2. Mobile app connects to that consumer +const wsUrl = buildAudioStreamUrl(consumer, token); +await audioStreamer.startStreaming(wsUrl, 'streaming'); + +// 3. Mobile app sends audio +recorder.startRecording((audioData) => { + audioStreamer.sendAudio(audioData); // Goes to Chronicle +}); +``` + +### Configuration Examples + +**Send to Chronicle** (default): +```yaml +selected_providers: + audio_consumer: chronicle +``` + +**Send to Mycelia**: +```yaml +selected_providers: + audio_consumer: mycelia +``` + +**Send to BOTH (multi-destination)**: +```yaml +selected_providers: + audio_consumer: multi-destination + +audio_consumer: + multi_dest_destinations: '[ + {"name":"chronicle","url":"ws://chronicle:5001/chronicle/ws_pcm"}, + {"name":"mycelia","url":"ws://mycelia:5173/ws_pcm"} + ]' +``` + +## Testing + +```bash +# Start backend +cd ushadow/backend +uvicorn main:app --reload + +# Test API +curl http://localhost:8000/api/providers/audio_consumer/active + +# Response: +{ + "capability": "audio_consumer", + "selected_provider": "chronicle", + "config": { + "provider_id": "chronicle", + "websocket_url": "ws://chronicle-backend:5001/chronicle/ws_pcm", + "protocol": "wyoming", + "format": "pcm_s16le_16khz_mono" + } +} + +# Switch to Mycelia +curl -X PUT http://localhost:8000/api/providers/audio_consumer/active \ + -H "Authorization: Bearer TOKEN" \ + -d '{"provider_id":"mycelia"}' +``` + +## Key Benefits + +✅ **Correct Semantics**: Audio sources are inputs, processors are consumers +✅ **Flexible Routing**: Any source → any consumer(s) +✅ **No Hardcoding**: Mobile app discovers consumer dynamically +✅ **Multi-Destination**: Built-in fanout support +✅ **Follows Pattern**: Same structure as LLM/transcription providers +✅ **Provider Discovery**: Mobile apps query API instead of hardcoded URLs + +## Next Steps + +1. **Configure default consumer** in `config/config.defaults.yaml` +2. **Mobile app integration** - Use `getActiveAudioConsumer()` to discover endpoint +3. **Test routing** - Send mobile audio to Chronicle, then switch to Mycelia +4. **Try multi-destination** - Send audio to both simultaneously diff --git a/compose/MYCELIA-INTEGRATION.md b/compose/MYCELIA-INTEGRATION.md new file mode 100644 index 00000000..8760b133 --- /dev/null +++ b/compose/MYCELIA-INTEGRATION.md @@ -0,0 +1,125 @@ +# Mycelia Integration with ushadow + +## Overview + +Mycelia has been integrated with ushadow's provider/instance model to support stateless configuration via environment variables. + +## Changes Made + +### 1. Schema Updates (`mycelia/myceliasdk/config.ts`) + +Updated the server config schema to support separate LLM and transcription providers: + +```typescript +export const zProviderConfig = z.object({ + baseUrl: z.string().optional(), + apiKey: z.string().optional(), + model: z.string().optional(), +}); + +export const zServerConfig = z.object({ + llm: zProviderConfig.optional().nullable(), // New: LLM-specific config + transcription: zProviderConfig.optional().nullable(), // New: Transcription-specific config + inference: zInferenceProviderConfig.optional().nullable(), // Deprecated: kept for backward compatibility + // ... +}); +``` + +### 2. Resource Updates + +Both `LLMResource` and `TranscriptionResource` now follow ushadow's stateless pattern: + +**Priority:** Environment variables → MongoDB (fallback) + +```typescript +async getInferenceProvider() { + // 1. Read from env vars (stateless - ushadow pattern) + const envBaseUrl = Deno.env.get("OPENAI_BASE_URL"); + const envApiKey = Deno.env.get("OPENAI_API_KEY"); + const envModel = Deno.env.get("OPENAI_MODEL"); + + if (envBaseUrl && envApiKey) { + return { baseUrl: envBaseUrl, apiKey: envApiKey, model: envModel }; + } + + // 2. Fallback to MongoDB for backward compatibility + const config = await getServerConfig(); + // ... +} +``` + +### 3. Compose File Configuration + +**Environment Variables** (compose/mycelia-compose.yml): + +```yaml +# LLM Provider Configuration +- OPENAI_BASE_URL=${OPENAI_BASE_URL} +- OPENAI_API_KEY=${OPENAI_API_KEY} +- OPENAI_MODEL=${OPENAI_MODEL} + +# Transcription Provider Configuration +- TRANSCRIPTION_BASE_URL=${TRANSCRIPTION_BASE_URL} +- TRANSCRIPTION_API_KEY=${TRANSCRIPTION_API_KEY} +- TRANSCRIPTION_MODEL=${TRANSCRIPTION_MODEL} +``` + +**ushadow Metadata:** + +```yaml +x-ushadow: + mycelia-backend: + requires: ["llm", "transcription"] # Declares capability requirements +``` + +## How It Works + +1. **Service Definition**: Mycelia declares it needs `llm` and `transcription` capabilities in x-ushadow metadata +2. **Provider Resolution**: ushadow's capability resolver maps these to provider instances +3. **Env Var Injection**: ushadow injects the mapped env vars into the container +4. **Runtime**: Mycelia reads configuration from env vars (stateless) + +## Backward Compatibility + +Mycelia maintains backward compatibility with its original MongoDB-based configuration: +- If env vars are not set, it falls back to reading from MongoDB +- Existing Mycelia installations continue to work unchanged +- The `inference` field is deprecated but still supported + +## Provider Requirements + +### LLM Provider +- **Base URL**: OpenAI-compatible API endpoint (e.g., http://ollama:11434/v1) +- **API Key**: Authentication key for the provider +- **Model**: Optional model name (e.g., "llama3") +- **Endpoint Used**: `/v1/chat/completions` + +### Transcription Provider +- **Base URL**: OpenAI-compatible Whisper API endpoint +- **API Key**: Authentication key for the provider +- **Model**: Optional model name (defaults to "whisper-1") +- **Endpoint Used**: `/v1/audio/transcriptions` + +## Example ushadow Provider Configuration + +```yaml +providers: + llm: + instances: + - id: ollama-local + base_url: http://ollama:11434/v1 + api_key: ollama + model: llama3 + + transcription: + instances: + - id: whisper-local + base_url: http://whisper:8000/v1 + api_key: whisper +``` + +When Mycelia is started, ushadow will: +1. Resolve `llm` → ollama-local instance +2. Resolve `transcription` → whisper-local instance +3. Inject env vars: `OPENAI_BASE_URL`, `OPENAI_API_KEY`, `TRANSCRIPTION_BASE_URL`, etc. +4. Start Mycelia with stateless configuration diff --git a/compose/agent-zero-compose.yaml b/compose/agent-zero-compose.yaml index ddea5718..d697e47c 100644 --- a/compose/agent-zero-compose.yaml +++ b/compose/agent-zero-compose.yaml @@ -52,7 +52,7 @@ services: - agent_zero_data:/a0 networks: - - infra-network + - ushadow-network # Enable access to host network for Ollama and other local services extra_hosts: @@ -72,6 +72,6 @@ volumes: name: ${COMPOSE_PROJECT_NAME:-ushadow}_agent_zero_data networks: - infra-network: + ushadow-network: external: true - name: ${COMPOSE_PROJECT_NAME:-ushadow}_infra-network + name: ushadow-network diff --git a/compose/backend.yml b/compose/backend.yml index 459bc084..e207c826 100644 --- a/compose/backend.yml +++ b/compose/backend.yml @@ -32,12 +32,13 @@ services: - ../ushadow/backend:/app - ../config:/config # Mount config directory (read-write for feature flags) - ../compose:/compose # Mount compose files for service management + - ../mycelia:/mycelia # Mount mycelia for building mycelia-backend service - /app/__pycache__ - /app/.pytest_cache # Docker socket for container management (Tailscale container control) - /var/run/docker.sock:/var/run/docker.sock networks: - - infra-network + - ushadow-network healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/health"] interval: 10s @@ -47,6 +48,6 @@ services: restart: unless-stopped networks: - infra-network: - name: infra-network + ushadow-network: + name: ushadow-network external: true diff --git a/compose/chronicle-compose.yaml b/compose/chronicle-compose.yaml index aeafd032..61cc2f67 100644 --- a/compose/chronicle-compose.yaml +++ b/compose/chronicle-compose.yaml @@ -10,7 +10,7 @@ x-ushadow: chronicle-backend: display_name: "Chronicle" description: "AI-powered voice journal and life logger with transcription and LLM analysis" - requires: [llm, transcription] + requires: [llm, transcription, audio_input] optional: [memory] # Uses memory if available, works without it route_path: /chronicle # Tailscale Serve route - all /chronicle/* requests go here chronicle-webui: @@ -72,7 +72,7 @@ services: - ${PROJECT_ROOT}/config/defaults.yml:/app/config/defaults.yml:ro networks: - - infra-network + - ushadow-network # NOTE: Depends on shared infrastructure services (mongo, redis, qdrant) # These must be started separately via docker-compose.infra.yml @@ -103,7 +103,7 @@ services: - VITE_BACKEND_URL=http://localhost:${CHRONICLE_PORT:-8080} - BACKEND_URL=http://chronicle-backend:8000 networks: - - infra-network + - ushadow-network depends_on: - chronicle-backend restart: unless-stopped @@ -119,6 +119,6 @@ volumes: # Use existing shared infrastructure network networks: - infra-network: - name: infra-network + ushadow-network: + name: ushadow-network external: true diff --git a/compose/docker-compose.infra.yml b/compose/docker-compose.infra.yml index 8ee3ab54..48078a77 100644 --- a/compose/docker-compose.infra.yml +++ b/compose/docker-compose.infra.yml @@ -17,16 +17,41 @@ services: - "27017:27017" volumes: - mongo_data:/data/db + command: ["--replSet", "rs0", "--bind_ip_all"] healthcheck: - test: ["CMD", "mongosh", "--quiet", "--eval", "db.adminCommand({ ping: 1 })"] + test: ["CMD", "mongosh", "--quiet", "--eval", "try { rs.status().ok } catch(e) { 0 }"] interval: 10s timeout: 5s retries: 5 - start_period: 10s + start_period: 15s networks: + - ushadow-network - infra-network restart: unless-stopped + # One-time replica set initialization (runs and exits) + mongo-init: + image: mongo:8.0 + container_name: mongo-init + profiles: ["infra"] + depends_on: + mongo: + condition: service_started + entrypoint: > + mongosh --host mongo --quiet --eval " + try { + rs.status(); + print('Replica set already initialized'); + } catch(e) { + print('Initializing replica set...'); + rs.initiate({_id: 'rs0', members: [{_id: 0, host: 'mongo:27017'}]}); + print('Replica set initialized'); + } + " + networks: + - infra-network + restart: "no" + redis: image: redis:7-alpine container_name: redis @@ -42,6 +67,7 @@ services: timeout: 3s retries: 5 networks: + - ushadow-network - infra-network restart: unless-stopped @@ -55,6 +81,7 @@ services: volumes: - qdrant_data:/qdrant/storage networks: + - ushadow-network - infra-network restart: unless-stopped healthcheck: @@ -80,6 +107,7 @@ services: - postgres_data:/var/lib/postgresql/data - ../config/postgres-init:/docker-entrypoint-initdb.d:ro networks: + - ushadow-network - infra-network restart: unless-stopped healthcheck: @@ -102,6 +130,7 @@ services: volumes: - neo4j_data:/data networks: + - ushadow-network - infra-network restart: unless-stopped @@ -128,11 +157,14 @@ services: # - NET_ADMIN # - NET_RAW # networks: - # - infra-network + # - ushadow-network # restart: unless-stopped # command: sh -c "tailscaled --tun=userspace-networking --statedir=/var/lib/tailscale & sleep infinity" networks: + ushadow-network: + name: ushadow-network + external: true infra-network: name: infra-network external: true diff --git a/compose/frontend.yml b/compose/frontend.yml index ff120f9c..b16ef8b7 100644 --- a/compose/frontend.yml +++ b/compose/frontend.yml @@ -12,12 +12,12 @@ services: - VITE_BACKEND_URL=${VITE_BACKEND_URL:-http://localhost:8000} - VITE_ENV_NAME=${VITE_ENV_NAME:-} networks: - - infra-network + - ushadow-network depends_on: - backend restart: unless-stopped networks: - infra-network: - name: infra-network + ushadow-network: + name: ushadow-network external: true diff --git a/compose/metamcp-compose.yaml b/compose/metamcp-compose.yaml index b00a5137..19334b3f 100644 --- a/compose/metamcp-compose.yaml +++ b/compose/metamcp-compose.yaml @@ -56,7 +56,7 @@ services: # Server configuration for auto-registration - ../config/metamcp:/app/config:ro networks: - - infra-network + - ushadow-network healthcheck: test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:12008/health"] interval: 30s @@ -66,8 +66,8 @@ services: restart: unless-stopped networks: - infra-network: - name: infra-network + ushadow-network: + name: ushadow-network external: true volumes: diff --git a/compose/mycelia-compose.yml b/compose/mycelia-compose.yml new file mode 100644 index 00000000..acb3daad --- /dev/null +++ b/compose/mycelia-compose.yml @@ -0,0 +1,169 @@ +# Mycelia - AI Memory and Timeline +# Self-hosted voice memo transcription, timeline, and memory system +# +# Usage: +# docker compose -f compose/mycelia-compose.yml up -d +# +# Access: +# - Web UI: http://localhost:${MYCELIA_FRONTEND_PORT:-18080} +# - Backend API: http://localhost:${MYCELIA_BACKEND_PORT:-15173} + +# ============================================================================= +# USHADOW METADATA (ignored by Docker, read by ushadow backend) +# ============================================================================= +x-ushadow: + namespace: mycelia + # Infra services to start before this compose (from docker-compose.infra.yml) + infra_services: ["mongo", "redis"] + mycelia-backend: + display_name: "Mycelia" + description: "Self-hosted AI memory and timeline - capture ideas via voice, screenshots, or text" + requires: ["llm", "transcription", "audio_input"] + provides: memory # Primary capability (timeline, transcription are secondary) + tags: ["ai", "memory", "voice", "transcription", "timeline"] + wizard: "mycelia" # ID of the setup wizard + mycelia-frontend: + display_name: "Mycelia Frontend" + description: "Mycelia web interface" + mycelia-python-worker: + display_name: "Mycelia Python Worker" + description: "Audio processing worker" + +services: + mycelia-frontend: + build: + context: ${PROJECT_ROOT:-..}/mycelia + dockerfile: ./frontend/Dockerfile.${MYCELIA_FRONTEND_MODE:-dev} + image: mycelia-frontend:latest + container_name: ${COMPOSE_PROJECT_NAME:-ushadow}-mycelia-frontend + ports: + - "${MYCELIA_FRONTEND_PORT:-18080}:8080" + volumes: + - ${PROJECT_ROOT:-..}/mycelia/frontend:/app + - ${PROJECT_ROOT:-..}/mycelia/myceliasdk:/app/myceliasdk + environment: + - MYCELIA_FRONTEND_MODE=dev + networks: + - ushadow-network + healthcheck: + test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8080"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 30s + restart: unless-stopped + + mycelia-backend: + build: + context: ${PROJECT_ROOT:-..}/mycelia + dockerfile: ./backend/Dockerfile + image: mycelia-backend:latest + container_name: ${COMPOSE_PROJECT_NAME:-ushadow}-mycelia-backend + ports: + - "${MYCELIA_BACKEND_PORT:-15173}:5173" + command: deno task dev + environment: + # Application URLs + - MYCELIA_URL=http://localhost:${MYCELIA_BACKEND_PORT:-15173} + - MYCELIA_BACKEND_INTERNAL_URL=http://mycelia-backend:5173 + - MYCELIA_FRONTEND_HOST=http://localhost:${MYCELIA_FRONTEND_PORT:-18080} + + # Authentication + - MYCELIA_TOKEN=${MYCELIA_TOKEN} + - MYCELIA_CLIENT_ID=${MYCELIA_CLIENT_ID} + - SECRET_KEY=${AUTH_SECRET_KEY} + + # Database Configuration - uses shared mongo from infra + - DATABASE_NAME=${MYCELIA_DATABASE_NAME:-mycelia} + - MONGO_URL=mongodb://mongo:27017/${MYCELIA_DATABASE_NAME:-mycelia}?directConnection=true + + # Redis Configuration - uses shared redis from infra + - REDIS_HOST=redis + - REDIS_PORT=6379 + - REDIS_PASSWORD=${REDIS_PASSWORD:-} + + # Python Worker + - PYTHON_WORKER_URL=http://mycelia-python-worker:8000 + + # LLM Provider Configuration + - OPENAI_BASE_URL=${OPENAI_BASE_URL} + - OPENAI_API_KEY=${OPENAI_API_KEY} + - OPENAI_MODEL=${OPENAI_MODEL} + + # Transcription Provider Configuration + - TRANSCRIPTION_BASE_URL=${TRANSCRIPTION_BASE_URL} + - TRANSCRIPTION_API_KEY=${TRANSCRIPTION_API_KEY} + - TRANSCRIPTION_MODEL=${TRANSCRIPTION_MODEL} + + # OpenTelemetry + - OTEL_DENO=${MYCELIA_OTEL_DENO:-false} + - OTEL_SERVICE_NAME=mycelia + - OTEL_SERVICE_VERSION=1.0.0 + - OTEL_EXPORTER_OTLP_ENDPOINT=${MYCELIA_OTEL_EXPORTER_OTLP_ENDPOINT:-http://localhost:4318} + + # Application Config + - LOG_LEVEL=${MYCELIA_LOG_LEVEL:-INFO} + - NODE_ENV=${NODE_ENV:-production} + - JOB_TRIGGERS_FAST=${JOB_TRIGGERS_FAST:-true} + volumes: + - ${PROJECT_ROOT:-..}/mycelia/backend:/app + - ${PROJECT_ROOT:-..}/mycelia/myceliasdk:/myceliasdk + networks: + - ushadow-network + healthcheck: + test: ["CMD", "deno", "eval", "const res = await fetch('http://localhost:5173/health'); Deno.exit(res.ok ? 0 : 1);"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 30s + deploy: + resources: + limits: + cpus: '2.0' + memory: 4G + reservations: + cpus: '0.2' + memory: 1G + restart: unless-stopped + + mycelia-python-worker: + build: + context: ${PROJECT_ROOT:-..}/mycelia + dockerfile: ./python/Dockerfile + image: mycelia-python:latest + container_name: ${COMPOSE_PROJECT_NAME:-ushadow}-mycelia-python-worker + environment: + - MYCELIA_URL=http://mycelia-backend:5173 + - SECRET_KEY=${AUTH_SECRET_KEY} + volumes: + - ${PROJECT_ROOT:-..}/mycelia/python:/app + - mycelia_worker_data:/root/.cache + networks: + - ushadow-network + depends_on: + mycelia-backend: + condition: service_healthy + healthcheck: + test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health').read()"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + deploy: + resources: + limits: + cpus: '2.0' + memory: 4G + reservations: + cpus: '0.5' + memory: 1G + restart: unless-stopped + +networks: + ushadow-network: + name: ushadow-network + external: true + +volumes: + mycelia_worker_data: + driver: local diff --git a/compose/openmemory-compose.yaml b/compose/openmemory-compose.yaml index 8d22210d..8b6117b6 100644 --- a/compose/openmemory-compose.yaml +++ b/compose/openmemory-compose.yaml @@ -52,7 +52,7 @@ services: volumes: - mem0_data:/app/data networks: - - infra-network + - ushadow-network healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8765/health"] interval: 10s @@ -70,14 +70,14 @@ services: - VITE_API_URL=http://localhost:${OPENMEMORY_PORT:-8765} - API_URL=http://mem0:8765 networks: - - infra-network + - ushadow-network depends_on: - mem0 restart: unless-stopped networks: - infra-network: - name: infra-network + ushadow-network: + name: ushadow-network external: true volumes: diff --git a/compose/parakeet-compose.yml b/compose/parakeet-compose.yml index 5d4a4149..add594b8 100644 --- a/compose/parakeet-compose.yml +++ b/compose/parakeet-compose.yml @@ -58,5 +58,5 @@ services: # Shared network for cross-project communication networks: default: - name: infra-network + name: ushadow-network external: true \ No newline at end of file diff --git a/compose/scripts/README.md b/compose/scripts/README.md new file mode 100644 index 00000000..c9dc145c --- /dev/null +++ b/compose/scripts/README.md @@ -0,0 +1,132 @@ +# Service Setup Scripts + +Utility scripts for service configuration and token generation. + +## Mycelia Token Generator + +Generate Mycelia authentication credentials without spinning up the full compose stack. + +### Python Script (Recommended) + +**Requirements:** +- Python 3.6+ +- `pymongo` library: `pip install pymongo` +- MongoDB running (either standalone or via ushadow's infra stack) + +**Usage:** + +```bash +# Basic usage (connects to localhost:27017) +python3 compose/scripts/mycelia-generate-token.py + +# Custom MongoDB URI +python3 compose/scripts/mycelia-generate-token.py --mongo-uri mongodb://localhost:27018 + +# Custom database name +python3 compose/scripts/mycelia-generate-token.py --db-name my_mycelia_db + +# See all options +python3 compose/scripts/mycelia-generate-token.py --help +``` + +**What it does:** +1. Connects to your MongoDB instance +2. Generates a cryptographically secure API key (`mycelia_...`) +3. Hashes the key and stores it in the `api_keys` collection +4. Returns both `MYCELIA_TOKEN` and `MYCELIA_CLIENT_ID` + +**Output:** +``` +✓ Credentials generated successfully! + +MYCELIA_CLIENT_ID=6967e390127eb6333b3d6e9e +MYCELIA_TOKEN=mycelia_baKIsM6qRqcG0WH29ZcqXVx8PYELgHcRlrCcUsDpcB4 +``` + +### Docker Compose Method + +If you don't have Python or prefer to use the official Mycelia tooling: + +```bash +docker compose -f compose/mycelia-compose.yml run --rm mycelia-backend \ + deno run -A server.ts token-create +``` + +This method requires: +- Mycelia backend image to be built +- MongoDB accessible from the container +- All Mycelia dependencies available + +### Bash Script (Advanced) + +For environments with `mongosh` installed: + +```bash +bash compose/scripts/mycelia-generate-token.sh +``` + +Falls back to docker compose if `mongosh` is not available. + +## Using Generated Credentials + +### Via ushadow Wizard + +1. Click "Setup" on the mycelia-backend service card +2. Run one of the commands above +3. Copy the `MYCELIA_TOKEN` and `MYCELIA_CLIENT_ID` values +4. Paste into the wizard form fields +5. Click "Save Credentials" + +The wizard will automatically save these to ushadow settings and inject them when starting Mycelia. + +### Manual Configuration + +Add to your `.env` file: + +```bash +MYCELIA_TOKEN=mycelia_baKIsM6qRqcG0WH29ZcqXVx8PYELgHcRlrCcUsDpcB4 +MYCELIA_CLIENT_ID=6967e390127eb6333b3d6e9e +``` + +Or add via ushadow settings API: + +```bash +curl -X PUT http://localhost:8360/api/settings \ + -H "Content-Type: application/json" \ + -d '{ + "mycelia.token": "mycelia_...", + "mycelia.client_id": "..." + }' +``` + +## Troubleshooting + +### Python script fails with "pymongo not installed" + +Install the required library: +```bash +pip install pymongo +``` + +### Cannot connect to MongoDB + +Make sure MongoDB is running: +```bash +# Check if running via docker +docker ps | grep mongo + +# Or start ushadow's infrastructure +docker compose -f compose/docker-compose.infra.yml up -d mongo +``` + +### Token already exists + +Each run creates a new token. You can have multiple active tokens. To revoke old tokens, use the Mycelia API or delete from the `api_keys` collection in MongoDB. + +## Security Notes + +- Tokens are cryptographically secure (32 random bytes) +- Tokens are hashed with SHA256 before storage +- Each token gets admin-level policies by default +- Store tokens securely - treat them like passwords +- Revoke unused tokens through the Mycelia admin interface diff --git a/compose/scripts/mycelia-generate-token.py b/compose/scripts/mycelia-generate-token.py new file mode 100755 index 00000000..c92c1695 --- /dev/null +++ b/compose/scripts/mycelia-generate-token.py @@ -0,0 +1,163 @@ +#!/usr/bin/env python3 +""" +Generate Mycelia authentication token and client ID + +This script creates API credentials directly in MongoDB without +needing to spin up the full Mycelia compose stack. + +Usage: + python3 mycelia-generate-token.py [--mongo-uri MONGO_URI] [--db-name DB_NAME] +""" + +import argparse +import base64 +import hashlib +import secrets +import sys +from datetime import datetime +from pathlib import Path + +try: + from pymongo import MongoClient + from bson import ObjectId + HAS_PYMONGO = True +except ImportError: + HAS_PYMONGO = False + + +def load_env_file(): + """Load .env file from project root if it exists.""" + env_vars = {} + env_file = Path(__file__).parent.parent.parent / '.env' + + if env_file.exists(): + with open(env_file) as f: + for line in f: + line = line.strip() + if line and not line.startswith('#') and '=' in line: + key, value = line.split('=', 1) + env_vars[key] = value.strip('"').strip("'") + + return env_vars + + +def generate_api_key(): + """Generate a random API key with mycelia_ prefix.""" + random_bytes = secrets.token_bytes(32) + return f"mycelia_{base64.urlsafe_b64encode(random_bytes).decode().rstrip('=')}" + + +def hash_api_key(api_key: str, salt: bytes) -> str: + """Hash an API key with salt using SHA256.""" + hasher = hashlib.sha256() + hasher.update(salt) + hasher.update(api_key.encode()) + return base64.b64encode(hasher.digest()).decode() + + +def generate_credentials_with_mongo(mongo_uri: str, db_name: str): + """Generate credentials and store in MongoDB.""" + if not HAS_PYMONGO: + print("ERROR: pymongo not installed. Install with: pip install pymongo") + print("Or use the docker compose method instead:") + print(" docker compose -f compose/mycelia-compose.yml run --rm mycelia-backend deno run -A server.ts token-create") + sys.exit(1) + + # Generate token and salt + api_key = generate_api_key() + salt = secrets.token_bytes(32) + hashed_key = hash_api_key(api_key, salt) + + # Connect to MongoDB + try: + client = MongoClient(mongo_uri, serverSelectionTimeoutMS=5000) + client.admin.command('ping') # Test connection + db = client[db_name] + + # Create API key document + doc = { + 'hashedKey': hashed_key, + 'salt': base64.b64encode(salt).decode(), + 'owner': 'admin', + 'name': f'ushadow_generated_{int(datetime.now().timestamp())}', + 'policies': [{'resource': '**', 'action': '**', 'effect': 'allow'}], + 'openPrefix': api_key[:16], + 'createdAt': datetime.now(), + 'isActive': True + } + + # Insert into database + result = db.api_keys.insert_one(doc) + client_id = str(result.inserted_id) + + # Print credentials + print("\n✓ Credentials generated successfully!\n") + print(f"MYCELIA_CLIENT_ID={client_id}") + print(f"MYCELIA_TOKEN={api_key}") + print("\nCopy these values into the ushadow wizard or your .env file") + + except Exception as e: + print(f"ERROR: Failed to connect to MongoDB: {e}") + print("\nAlternatively, use the docker compose method:") + print(" docker compose -f compose/mycelia-compose.yml run --rm mycelia-backend deno run -A server.ts token-create") + sys.exit(1) + + +def main(): + parser = argparse.ArgumentParser( + description='Generate Mycelia authentication credentials', + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Examples: + # Use default MongoDB connection (localhost:27017) + python3 mycelia-generate-token.py + + # Specify custom MongoDB URI + python3 mycelia-generate-token.py --mongo-uri mongodb://localhost:27018 + + # Specify custom database name + python3 mycelia-generate-token.py --db-name my_mycelia_db + +If pymongo is not installed or MongoDB is not accessible, the script will +provide instructions to use the docker compose method instead. + """ + ) + + parser.add_argument( + '--mongo-uri', + default=None, + help='MongoDB connection URI (default: mongodb://localhost:27017)' + ) + + parser.add_argument( + '--db-name', + default='mycelia', + help='Database name (default: mycelia)' + ) + + args = parser.parse_args() + + # Load environment variables + env_vars = load_env_file() + + # Determine MongoDB URI + if args.mongo_uri: + mongo_uri = args.mongo_uri + elif 'MONGO_URL' in env_vars: + mongo_uri = env_vars['MONGO_URL'] + else: + mongo_uri = 'mongodb://localhost:27017' + + # Determine database name + db_name = env_vars.get('MYCELIA_DATABASE_NAME', args.db_name) + + print("Mycelia Token Generator") + print("=======================\n") + print(f"MongoDB URI: {mongo_uri}") + print(f"Database: {db_name}\n") + + generate_credentials_with_mongo(mongo_uri, db_name) + + +if __name__ == '__main__': + main() diff --git a/compose/scripts/mycelia-generate-token.sh b/compose/scripts/mycelia-generate-token.sh new file mode 100755 index 00000000..4d90f929 --- /dev/null +++ b/compose/scripts/mycelia-generate-token.sh @@ -0,0 +1,65 @@ +#!/bin/bash +# Generate Mycelia authentication token and client ID +# This script connects to MongoDB and creates the credentials directly + +set -e + +# Colors for output +GREEN='\033[0;32m' +BLUE='\033[0;34m' +YELLOW='\033[1;33m' +NC='\033[0m' # No Color + +echo -e "${BLUE}Mycelia Token Generator${NC}" +echo "========================" +echo + +# Check if mongosh is available +if ! command -v mongosh &> /dev/null; then + echo -e "${YELLOW}Warning: mongosh not found. Falling back to docker compose method...${NC}" + echo + docker compose -f compose/mycelia-compose.yml run --rm mycelia-backend deno run -A server.ts token-create + exit 0 +fi + +# Read MongoDB connection info from .env or use defaults +MONGO_HOST=${MONGO_HOST:-localhost} +MONGO_PORT=${MONGO_PORT:-27017} +MONGO_DB=${MYCELIA_DATABASE_NAME:-mycelia} + +# Generate random token +TOKEN="mycelia_$(openssl rand -base64 32 | tr -d '/+=' | head -c 43)" + +# Generate salt and hash +SALT=$(openssl rand -base64 32) +SALT_HEX=$(echo -n "$SALT" | base64 -d | xxd -p | tr -d '\n') +HASH=$(echo -n "${SALT_HEX}${TOKEN}" | xxd -r -p | openssl sha256 -binary | base64) + +# Create MongoDB document +TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%S.000Z") + +# Insert into MongoDB and capture the ID +MONGO_SCRIPT=" +db = db.getSiblingDB('${MONGO_DB}'); +var result = db.api_keys.insertOne({ + hashedKey: '${HASH}', + salt: '${SALT}', + owner: 'admin', + name: 'ushadow_generated_$(date +%s)', + policies: [{ resource: '**', action: '**', effect: 'allow' }], + openPrefix: '${TOKEN:0:16}', + createdAt: new Date('${TIMESTAMP}'), + isActive: true +}); +print(result.insertedId.toString()); +" + +CLIENT_ID=$(mongosh --quiet --host "${MONGO_HOST}" --port "${MONGO_PORT}" --eval "${MONGO_SCRIPT}") + +# Output credentials +echo -e "${GREEN}✓ Credentials generated successfully!${NC}" +echo +echo -e "${BLUE}MYCELIA_CLIENT_ID=${NC}${CLIENT_ID}" +echo -e "${BLUE}MYCELIA_TOKEN=${NC}${TOKEN}" +echo +echo -e "${YELLOW}Copy these values into the ushadow wizard or your .env file${NC}" diff --git a/compose/speaker-recognition-compose.yaml b/compose/speaker-recognition-compose.yaml index 29989738..41c37cc2 100644 --- a/compose/speaker-recognition-compose.yaml +++ b/compose/speaker-recognition-compose.yaml @@ -53,7 +53,7 @@ services: - SPEAKER_SERVICE_PORT=${SPEAKER_SERVICE_PORT:-8085} - DEEPGRAM_API_KEY=${DEEPGRAM_API_KEY:-} networks: - - infra-network + - ushadow-network restart: unless-stopped healthcheck: test: ["CMD", "curl", "-f", "http://localhost:${SPEAKER_SERVICE_PORT:-8085}/health"] @@ -90,7 +90,7 @@ services: default: aliases: - speaker-recognition - infra-network: + ushadow-network: aliases: - speaker-recognition restart: unless-stopped @@ -132,7 +132,7 @@ services: - SPEAKER_SERVICE_PORT=${SPEAKER_SERVICE_PORT:-8085} - VITE_SPEAKER_SERVICE_URL=http://localhost:${SPEAKER_SERVICE_PORT:-8085} networks: - - infra-network + - ushadow-network # Note: No depends_on - both cpu and gpu services use 'speaker-recognition' network alias # The webui connects by alias and handles connection retries gracefully restart: unless-stopped @@ -158,7 +158,7 @@ services: depends_on: - speaker-recognition-webui networks: - - infra-network + - ushadow-network restart: unless-stopped # ============================================================================= @@ -178,6 +178,6 @@ volumes: # Networks # ============================================================================= networks: - infra-network: - name: infra-network + ushadow-network: + name: ushadow-network external: true diff --git a/compose/tailscale-compose.yml b/compose/tailscale-compose.yml new file mode 100644 index 00000000..5ca260af --- /dev/null +++ b/compose/tailscale-compose.yml @@ -0,0 +1,54 @@ +# Tailscale Serve Proxy +# +# Provides HTTPS ingress via Tailscale Serve, proxying to internal services. +# Must be on ushadow-network to reach backend/webui containers. +# +# Usage: +# docker compose -f compose/tailscale-compose.yml up -d +# +# Setup: +# 1. Start container +# 2. Run: docker exec tailscale tailscale login +# 3. Authenticate via URL +# 4. Backend auto-configures Serve routes via /api/tailscale/serve-config + +x-ushadow: + tailscale: + display_name: "Tailscale" + description: "Tailscale Serve HTTPS proxy for secure external access" + requires: [] + provides: tailscale + tags: ["networking", "proxy", "security"] + +services: + tailscale: + image: tailscale/tailscale:latest + container_name: ${COMPOSE_PROJECT_NAME:-ushadow}-tailscale + hostname: ${COMPOSE_PROJECT_NAME:-ushadow}-tailscale + environment: + - TS_STATE_DIR=/var/lib/tailscale + - TS_USERSPACE=true + - TS_ACCEPT_DNS=true + - TS_EXTRA_ARGS=--advertise-tags=tag:container + volumes: + - tailscale_state:/var/lib/tailscale + cap_add: + - NET_ADMIN + - NET_RAW + networks: + - ushadow-network + - infra-network + restart: unless-stopped + command: sh -c "tailscaled --tun=userspace-networking --statedir=/var/lib/tailscale & sleep 2 && tailscale up --hostname=${COMPOSE_PROJECT_NAME:-ushadow} && sleep infinity" + +networks: + ushadow-network: + name: ushadow-network + external: true + infra-network: + name: infra-network + external: true + +volumes: + tailscale_state: + driver: local diff --git a/compose/whisper-compose.yml b/compose/whisper-compose.yml new file mode 100644 index 00000000..bdf82a47 --- /dev/null +++ b/compose/whisper-compose.yml @@ -0,0 +1,38 @@ +x-ushadow: + faster-whisper: + display_name: "Faster Whisper" + description: "Local speech-to-text with OpenAI-compatible API" + requires: [] + provides: transcription + tags: ["audio", "transcription", "local"] + +services: + faster-whisper: + image: fedirz/faster-whisper-server:latest-cpu + container_name: faster-whisper + environment: + - WHISPER__MODEL=Systran/faster-whisper-base + - UVICORN_HOST=0.0.0.0 + - UVICORN_PORT=8000 + volumes: + - whisper_models:/root/.cache/huggingface + ports: + - "${WHISPER_PORT:-10300}:8000" + networks: + - ushadow-network + restart: unless-stopped + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + +networks: + ushadow-network: + name: ushadow-network + external: true + +volumes: + whisper_models: + driver: local \ No newline at end of file diff --git a/config/capabilities.yaml b/config/capabilities.yaml index e6d2d1d2..f95f64df 100644 --- a/config/capabilities.yaml +++ b/config/capabilities.yaml @@ -94,3 +94,40 @@ capabilities: sync_interval: type: integer description: "Auto-sync interval in seconds" + + # ========================================================================== + # Audio Input Capability (Audio Sources) + # ========================================================================== + audio_input: + description: "Audio input sources that provide audio streams" + + provides: + device_type: + type: string + description: "Type of audio input device (mobile, omi, desktop, file)" + format: + type: string + description: "Audio format specification (pcm_s16le_16khz_mono)" + protocol: + type: string + description: "Streaming protocol used (wyoming, webrtc, raw)" + + # ========================================================================== + # Audio Consumer Capability (Audio Processing Services) + # ========================================================================== + audio_consumer: + description: "Services that consume and process audio streams" + + provides: + websocket_url: + type: url + description: "WebSocket endpoint that accepts audio" + protocol: + type: string + description: "Audio streaming protocol (wyoming, webrtc, raw)" + api_key: + type: secret + description: "API key or JWT token (if required)" + format: + type: string + description: "Accepted audio format" diff --git a/config/config.defaults.yaml b/config/config.defaults.yaml index dc15504f..f5d91e8b 100644 --- a/config/config.defaults.yaml +++ b/config/config.defaults.yaml @@ -110,9 +110,11 @@ wizard: # Selected Providers per Capability # Services declare `uses: [{capability: llm}]` and get the selected provider's env vars. selected_providers: - llm: openai # openai | anthropic | ollama | openai-compatible - transcription: deepgram # deepgram | mistral-voxtral | whisper-local | parakeet - memory: openmemory # openmemory | cognee | mem0-cloud + llm: openai # openai | anthropic | ollama | openai-compatible + transcription: deepgram # deepgram | mistral-voxtral | whisper-local | parakeet + memory: openmemory # openmemory | cognee | mem0-cloud + audio_input: mobile-app # mobile-app | omi-device | desktop-mic | audio-file | unode + audio_consumer: chronicle # chronicle | mycelia | multi-destination | custom-websocket | webhook # Default Services to Install # These are installed by default in quickstart mode. User modifications diff --git a/config/providers/audio_consumer.yaml b/config/providers/audio_consumer.yaml new file mode 100644 index 00000000..16bb2962 --- /dev/null +++ b/config/providers/audio_consumer.yaml @@ -0,0 +1,249 @@ +# Audio Consumer Providers (Audio Processing Services) +# Implements the 'audio_consumer' capability defined in capabilities.yaml +# +# Audio consumers are SERVICES that receive and process audio +# They CONSUME audio from audio inputs (mobile, Omi, etc.) + +capability: audio_consumer + +providers: + # ========================================================================== + # Chronicle - Audio Transcription & Conversation Tracking + # ========================================================================== + - id: chronicle + name: "Chronicle" + description: "Audio transcription, speaker diarization, and conversation management" + mode: local + + docker: + image: ushadow/chronicle-backend:latest + compose_file: compose/chronicle-compose.yaml + service_name: chronicle-backend + ports: + - container: 5001 + protocol: http + health: + http_get: /health + port: 5001 + + credentials: + websocket_url: + env_var: CHRONICLE_WS_URL + settings_path: audio_consumer.chronicle_ws_url + label: "Chronicle WebSocket URL" + default: "ws://chronicle-backend:5001/chronicle/ws_pcm" + protocol: + env_var: CHRONICLE_PROTOCOL + value: "wyoming" + api_key: + env_var: CHRONICLE_API_KEY + settings_path: audio_consumer.chronicle_api_key + label: "JWT Token" + required: false + format: + env_var: CHRONICLE_AUDIO_FORMAT + value: "pcm_s16le_16khz_mono" + + config: + recording_mode: + env_var: CHRONICLE_RECORDING_MODE + settings_path: audio_consumer.chronicle_mode + label: "Recording Mode" + description: "batch = process after completion, streaming = real-time" + default: "streaming" + options: ["streaming", "batch"] + enable_diarization: + env_var: CHRONICLE_ENABLE_DIARIZATION + settings_path: audio_consumer.chronicle_diarization + label: "Enable Speaker Diarization" + default: true + + capabilities: + - transcription + - speaker_diarization + - conversation_tracking + - memory_integration + + ui: + icon: chronicle + tags: ["audio", "consumer", "transcription", "local"] + + # ========================================================================== + # Mycelia - Audio Processing & Workflow Engine + # ========================================================================== + - id: mycelia + name: "Mycelia" + description: "Audio processing, timeline storage, and custom workflow orchestration" + mode: local + + docker: + image: mycelia/backend:latest + compose_file: "" + service_name: mycelia-backend + ports: + - container: 5173 + protocol: http + health: + http_get: /health + port: 5173 + + credentials: + websocket_url: + env_var: MYCELIA_WS_URL + settings_path: audio_consumer.mycelia_ws_url + label: "Mycelia WebSocket URL" + default: "ws://mycelia-backend:5173/ws_pcm" + protocol: + env_var: MYCELIA_PROTOCOL + value: "wyoming" + api_key: + env_var: MYCELIA_API_KEY + settings_path: audio_consumer.mycelia_api_key + label: "JWT Token" + required: true + format: + env_var: MYCELIA_AUDIO_FORMAT + value: "pcm_s16le_16khz_mono" + + config: + processing_mode: + env_var: MYCELIA_PROCESSING_MODE + settings_path: audio_consumer.mycelia_mode + label: "Processing Mode" + default: "streaming" + options: ["streaming", "batch"] + chunk_duration: + env_var: MYCELIA_CHUNK_DURATION + settings_path: audio_consumer.mycelia_chunk_duration + label: "Chunk Duration (seconds)" + default: 10 + + capabilities: + - audio_storage + - timeline_management + - workflow_processing + - audio_chunking + + ui: + icon: mycelia + tags: ["audio", "consumer", "processing", "local"] + + # ========================================================================== + # Multi-Destination (Fanout to Multiple Consumers) + # ========================================================================== + - id: multi-destination + name: "Multi-Destination" + description: "Send audio to multiple consumers simultaneously (Chronicle + Mycelia + more)" + mode: relay + + credentials: + websocket_url: + env_var: MULTI_DEST_WS_URL + settings_path: audio_consumer.multi_dest_ws_url + label: "Relay WebSocket URL" + default: "ws://ushadow-backend:8000/ws/audio/relay" + protocol: + env_var: MULTI_DEST_PROTOCOL + value: "wyoming" + api_key: + env_var: MULTI_DEST_API_KEY + settings_path: audio_consumer.multi_dest_api_key + label: "JWT Token" + required: true + format: + env_var: MULTI_DEST_AUDIO_FORMAT + value: "pcm_s16le_16khz_mono" + + config: + destinations: + env_var: MULTI_DEST_DESTINATIONS + settings_path: audio_consumer.multi_dest_destinations + label: "Destinations" + description: "JSON array of {name, url} consumer endpoints" + default: '[{"name":"chronicle","url":"ws://chronicle-backend:5001/chronicle/ws_pcm"},{"name":"mycelia","url":"ws://mycelia-backend:5173/ws_pcm"}]' + type: json + + capabilities: + - fanout + - load_balancing + - redundancy + + ui: + icon: relay + tags: ["audio", "consumer", "relay", "fanout"] + + # ========================================================================== + # Custom WebSocket Consumer + # ========================================================================== + - id: custom-websocket + name: "Custom WebSocket" + description: "Connect to any custom audio processing endpoint" + mode: custom + + credentials: + websocket_url: + env_var: CUSTOM_CONSUMER_WS_URL + settings_path: audio_consumer.custom_ws_url + label: "WebSocket URL" + required: true + placeholder: "ws://your-server:port/audio/input" + protocol: + env_var: CUSTOM_CONSUMER_PROTOCOL + settings_path: audio_consumer.custom_protocol + label: "Protocol" + default: "wyoming" + options: ["wyoming", "raw", "webrtc", "custom"] + api_key: + env_var: CUSTOM_CONSUMER_API_KEY + settings_path: audio_consumer.custom_api_key + label: "API Key/Token" + required: false + format: + env_var: CUSTOM_CONSUMER_AUDIO_FORMAT + settings_path: audio_consumer.custom_format + label: "Audio Format" + default: "pcm_s16le_16khz_mono" + + ui: + icon: custom + tags: ["audio", "consumer", "custom", "advanced"] + + # ========================================================================== + # Webhook Consumer (HTTP POST) + # ========================================================================== + - id: webhook + name: "Webhook" + description: "POST audio to HTTP endpoint for external processing" + mode: webhook + + credentials: + websocket_url: + env_var: WEBHOOK_URL + settings_path: audio_consumer.webhook_url + label: "Webhook URL" + required: true + placeholder: "https://your-api.com/audio/process" + protocol: + env_var: WEBHOOK_PROTOCOL + value: "http" + api_key: + env_var: WEBHOOK_API_KEY + settings_path: audio_consumer.webhook_api_key + label: "API Key" + required: false + format: + env_var: WEBHOOK_AUDIO_FORMAT + settings_path: audio_consumer.webhook_format + label: "Audio Format" + default: "wav" + options: ["wav", "mp3", "opus", "raw"] + + config: + batch_size: + label: "Batch Size" + description: "Number of audio chunks to batch before sending" + default: 1 + + ui: + icon: webhook + tags: ["audio", "consumer", "webhook", "http"] diff --git a/config/providers/audio_input.yaml b/config/providers/audio_input.yaml new file mode 100644 index 00000000..66d11fcc --- /dev/null +++ b/config/providers/audio_input.yaml @@ -0,0 +1,146 @@ +# Audio Input Providers (Audio Sources) +# Implements the 'audio_input' capability defined in capabilities.yaml +# +# Audio inputs are SOURCES of audio (mobile app, Omi device, desktop mic, etc.) +# They PROVIDE audio that gets streamed TO audio consumers (Chronicle, Mycelia) + +capability: audio_input + +providers: + # ========================================================================== + # Mobile App Microphone + # ========================================================================== + - id: mobile-app + name: "Mobile App" + description: "Audio from mobile device microphone (iOS/Android)" + mode: client + + credentials: + device_type: + value: "mobile" + format: + value: "pcm_s16le_16khz_mono" + protocol: + value: "wyoming" + + config: + sampling_rate: + label: "Sampling Rate" + default: 16000 + options: [8000, 16000, 44100, 48000] + channels: + label: "Channels" + default: 1 + options: [1, 2] + bit_depth: + label: "Bit Depth" + default: 16 + options: [16, 24, 32] + + ui: + icon: phone + tags: ["audio", "mobile", "mic", "client"] + + # ========================================================================== + # Omi Device (Bluetooth Audio) + # ========================================================================== + - id: omi-device + name: "Omi Device" + description: "Omi wearable audio recording device via Bluetooth" + mode: client + + credentials: + device_type: + value: "omi" + format: + value: "pcm_s16le_16khz_mono" + protocol: + value: "wyoming" + + config: + codec: + label: "Bluetooth Codec" + description: "Codec used for Bluetooth transmission" + default: "opus" + options: ["opus", "aac", "sbc"] + + ui: + icon: bluetooth + tags: ["audio", "omi", "bluetooth", "wearable"] + + # ========================================================================== + # Desktop Microphone + # ========================================================================== + - id: desktop-mic + name: "Desktop Microphone" + description: "System microphone on desktop/laptop" + mode: client + + credentials: + device_type: + value: "desktop" + format: + value: "pcm_s16le_16khz_mono" + protocol: + value: "wyoming" + + config: + device_id: + label: "Audio Device" + description: "Select system audio input device" + type: select + + ui: + icon: microphone + tags: ["audio", "desktop", "mic", "client"] + + # ========================================================================== + # Audio File Upload + # ========================================================================== + - id: audio-file + name: "Audio File" + description: "Upload pre-recorded audio files for processing" + mode: upload + + credentials: + device_type: + value: "file" + format: + value: "auto" # Auto-detect from file + protocol: + value: "http" # File upload via HTTP POST + + config: + supported_formats: + label: "Supported Formats" + value: ["wav", "mp3", "m4a", "flac", "ogg", "opus"] + + ui: + icon: upload + tags: ["audio", "file", "upload", "batch"] + + # ========================================================================== + # UNode (Remote Audio Device) + # ========================================================================== + - id: unode + name: "UNode Device" + description: "Remote audio streaming node (Raspberry Pi, server, etc.)" + mode: remote + + credentials: + device_type: + value: "unode" + format: + value: "pcm_s16le_16khz_mono" + protocol: + value: "wyoming" + + config: + device_name: + label: "Device Name" + description: "Friendly name for this UNode" + type: string + + ui: + icon: server + tags: ["audio", "unode", "remote", "iot"] diff --git a/docs/AUDIO_PROVIDER_ARCHITECTURE.md b/docs/AUDIO_PROVIDER_ARCHITECTURE.md new file mode 100644 index 00000000..02e0c9a9 --- /dev/null +++ b/docs/AUDIO_PROVIDER_ARCHITECTURE.md @@ -0,0 +1,313 @@ +# Audio Provider Architecture + +## Correct Architecture + +``` +Audio INPUT Providers (Sources) Audio CONSUMERS (Processing Services) + ├─ Mobile App Mic ─────→ ├─ Chronicle + ├─ Omi Device ─────→ ├─ Mycelia + ├─ Desktop Mic ─────→ ├─ Multi-Destination + ├─ Audio File Upload ─────→ └─ Custom Webhook + └─ UNode Device ─────→ +``` + +## Two Separate Capabilities + +### 1. `audio_input` - Audio Sources (Providers) +**What they do**: Generate/capture audio and stream it out +**Examples**: Mobile app, Omi device, desktop microphone, file upload +**Config file**: `config/providers/audio_input.yaml` + +**Available Providers**: +- `mobile-app` - Mobile device microphone (iOS/Android) +- `omi-device` - Omi wearable Bluetooth device +- `desktop-mic` - Desktop/laptop microphone +- `audio-file` - Pre-recorded audio file upload +- `unode` - Remote audio streaming node (Raspberry Pi, etc.) + +### 2. `audio_consumer` - Audio Processing Services (Consumers) +**What they do**: Receive audio streams and process them +**Examples**: Chronicle (transcription), Mycelia (processing), custom webhooks +**Config file**: `config/providers/audio_consumer.yaml` + +**Available Providers**: +- `chronicle` - Transcription, speaker diarization, conversation tracking (default) +- `mycelia` - Audio processing, timeline storage, workflow orchestration +- `multi-destination` - Fanout to multiple consumers simultaneously +- `custom-websocket` - Any custom WebSocket endpoint +- `webhook` - HTTP POST to external API + +## Configuration + +### Default Selection +**File**: `config/config.defaults.yaml` + +```yaml +selected_providers: + audio_input: mobile-app # Which audio source + audio_consumer: chronicle # Where audio goes +``` + +### Routing: Mobile App → Chronicle + +```yaml +# Mobile app streams audio to Chronicle +audio_input: mobile-app +audio_consumer: chronicle +``` + +Mobile app connects to Chronicle's WebSocket endpoint. + +### Routing: Mobile App → Mycelia + +```yaml +# Mobile app streams audio to Mycelia instead +audio_input: mobile-app +audio_consumer: mycelia +``` + +Mobile app connects to Mycelia's WebSocket endpoint. + +### Routing: Mobile App → Both (Multi-Destination) + +```yaml +# Mobile app streams to BOTH Chronicle and Mycelia +audio_input: mobile-app +audio_consumer: multi-destination + +# Configure destinations +audio_consumer: + multi_dest_destinations: '[ + {"name":"chronicle","url":"ws://chronicle-backend:5001/chronicle/ws_pcm"}, + {"name":"mycelia","url":"ws://mycelia-backend:5173/ws_pcm"} + ]' +``` + +Mobile app connects to relay server, which fans out to both consumers. + +## How It Works + +### Step 1: Audio Input Provides Audio +```typescript +// Mobile app (audio input provider) +const audioStreamer = useAudioStreamer(); +const recorder = usePhoneAudioRecorder(); + +// Start recording +await recorder.startRecording((audioData) => { + audioStreamer.sendAudio(new Uint8Array(audioData)); +}); +``` + +### Step 2: Get Audio Consumer Config +```typescript +// Mobile app fetches where to send audio +const consumer = await getActiveAudioConsumer(baseUrl, token); + +// Returns: +// { +// provider_id: "chronicle", +// websocket_url: "ws://chronicle-backend:5001/chronicle/ws_pcm", +// protocol: "wyoming", +// format: "pcm_s16le_16khz_mono" +// } +``` + +### Step 3: Connect to Consumer +```typescript +// Mobile app connects to the selected consumer +const wsUrl = buildAudioStreamUrl(consumer, token); +await audioStreamer.startStreaming(wsUrl, 'streaming'); + +// Now audio flows: Mobile Mic → Chronicle +``` + +## Example Configurations + +### Configuration 1: Mobile → Chronicle (Default) +```yaml +audio_input: mobile-app +audio_consumer: chronicle +``` + +**Flow**: Mobile microphone → Chronicle transcription +**Use case**: Standard transcription and conversation tracking + +### Configuration 2: Omi Device → Mycelia +```yaml +audio_input: omi-device +audio_consumer: mycelia +``` + +**Flow**: Omi Bluetooth device → Mycelia processing +**Use case**: Wearable audio → custom processing workflows + +### Configuration 3: Desktop → Multi-Destination +```yaml +audio_input: desktop-mic +audio_consumer: multi-destination + +audio_consumer: + multi_dest_destinations: '[ + {"name":"chronicle","url":"ws://chronicle:5001/chronicle/ws_pcm"}, + {"name":"mycelia","url":"ws://mycelia:5173/ws_pcm"} + ]' +``` + +**Flow**: Desktop mic → Relay → Chronicle + Mycelia +**Use case**: Send audio to multiple processors simultaneously + +### Configuration 4: File Upload → Webhook +```yaml +audio_input: audio-file +audio_consumer: webhook + +audio_consumer: + webhook_url: "https://api.external.com/audio/process" + webhook_api_key: "your-api-key" +``` + +**Flow**: Audio file → External API +**Use case**: Batch processing of audio files via external service + +## API Endpoints + +### Get Active Audio Consumer +```bash +GET /api/providers/audio_consumer/active + +Response: +{ + "provider_id": "chronicle", + "websocket_url": "ws://chronicle-backend:5001/chronicle/ws_pcm", + "protocol": "wyoming", + "format": "pcm_s16le_16khz_mono" +} +``` + +### Get Available Audio Consumers +```bash +GET /api/providers/audio_consumer/available + +Response: +{ + "providers": [ + {"id": "chronicle", "name": "Chronicle", "mode": "local"}, + {"id": "mycelia", "name": "Mycelia", "mode": "local"}, + {"id": "multi-destination", "name": "Multi-Destination", "mode": "relay"} + ] +} +``` + +### Switch Audio Consumer +```bash +PUT /api/providers/audio_consumer/active +{ + "provider_id": "mycelia" +} + +Response: +{ + "success": true, + "selected_provider": "mycelia", + "message": "Audio consumer set to 'Mycelia'" +} +``` + +## Mobile App Integration + +The mobile app automatically discovers and connects to the selected audio consumer: + +```typescript +import { getActiveAudioConsumer, buildAudioStreamUrl } from './services/audioProviderApi'; + +// 1. Fetch active consumer configuration +const consumer = await getActiveAudioConsumer(baseUrl, jwtToken); + +// 2. Build WebSocket URL with authentication +const wsUrl = buildAudioStreamUrl(consumer, jwtToken); +// Result: "ws://chronicle:5001/chronicle/ws_pcm?token=JWT_HERE" + +// 3. Start streaming to consumer +await audioStreamer.startStreaming(wsUrl, 'streaming'); + +// 4. Send audio data +recorder.startRecording((audioData) => { + audioStreamer.sendAudio(new Uint8Array(audioData)); +}); +``` + +**No hardcoded URLs!** Mobile app dynamically discovers where to send audio based on server configuration. + +## Switching Consumers + +### From Chronicle to Mycelia + +**Before**: +```yaml +audio_consumer: chronicle +``` + +Mobile app streams to Chronicle endpoint. + +**After**: +```yaml +audio_consumer: mycelia +``` + +Mobile app now streams to Mycelia endpoint (after refresh/reconnect). + +### From Single to Multi-Destination + +**Before**: +```yaml +audio_consumer: chronicle +``` + +**After**: +```yaml +audio_consumer: multi-destination + +audio_consumer: + multi_dest_destinations: '[ + {"name":"chronicle","url":"ws://chronicle:5001/chronicle/ws_pcm"}, + {"name":"mycelia","url":"ws://mycelia:5173/ws_pcm"} + ]' +``` + +Mobile app connects once to relay, audio goes to both Chronicle AND Mycelia. + +## Benefits + +✅ **Separation of Concerns**: Input sources and processing services are separate +✅ **Flexible Routing**: Route any input to any consumer(s) +✅ **No Hardcoded URLs**: Mobile app discovers consumer endpoints dynamically +✅ **Multi-Destination**: Built-in fanout to multiple consumers +✅ **Easy Switching**: Change consumer in config, no app changes needed + +## Wiring Examples + +For advanced routing, use `config/wiring.yaml`: + +```yaml +wiring: + # Mobile app audio → Chronicle + - source_instance_id: mobile-app-1 + source_capability: audio_input + target_instance_id: chronicle-compose:chronicle-backend + target_capability: audio_consumer + + # Omi device audio → Mycelia + - source_instance_id: omi-device-1 + source_capability: audio_input + target_instance_id: mycelia-compose:mycelia-backend + target_capability: audio_consumer + + # Desktop audio → Multi-destination (Chronicle + Mycelia) + - source_instance_id: desktop-mic-1 + source_capability: audio_input + target_instance_id: audio-relay + target_capability: audio_consumer +``` + +This allows different audio sources to route to different consumers! diff --git a/ushadow/backend/main.py b/ushadow/backend/main.py index af593793..fdb133c0 100644 --- a/ushadow/backend/main.py +++ b/ushadow/backend/main.py @@ -22,8 +22,8 @@ from src.routers import health, wizard, chronicle, auth, feature_flags from src.routers import services, deployments, providers, service_configs, chat -from src.routers import kubernetes, tailscale, unodes, docker -from src.routers import github_import +from src.routers import kubernetes, tailscale, unodes, docker, sse +from src.routers import github_import, audio_relay, audio_provider from src.routers import settings as settings_api from src.middleware import setup_middleware from src.services.unode_manager import init_unode_manager, get_unode_manager @@ -181,7 +181,10 @@ def send_telemetry(): app.include_router(chat.router, prefix="/api/chat", tags=["chat"]) app.include_router(deployments.router, tags=["deployments"]) app.include_router(tailscale.router, tags=["tailscale"]) +app.include_router(sse.router, prefix="/api/sse", tags=["sse"]) app.include_router(github_import.router, prefix="/api/github-import", tags=["github-import"]) +app.include_router(audio_relay.router, tags=["audio"]) +app.include_router(audio_provider.router, tags=["providers"]) # Setup MCP server for LLM tool access setup_mcp_server(app) diff --git a/ushadow/backend/src/config/omegaconf_settings.py b/ushadow/backend/src/config/omegaconf_settings.py index 616a03a1..1c97ad99 100644 --- a/ushadow/backend/src/config/omegaconf_settings.py +++ b/ushadow/backend/src/config/omegaconf_settings.py @@ -119,9 +119,9 @@ def to_dict(self) -> Dict[str, Any]: # Sections to search for different setting types SETTING_SECTIONS = { - 'secret': ['api_keys', 'security', 'admin'], + 'secret': ['api_keys', 'security', 'admin', 'services', 'mycelia', 'settings'], 'url': ['services'], - 'string': ['llm', 'transcription', 'memory', 'auth', 'security', 'admin'], + 'string': ['llm', 'transcription', 'memory', 'auth', 'security', 'admin', 'mycelia', 'settings'], } # ============================================================================= @@ -220,12 +220,53 @@ def __init__(self, config_dir: Optional[Path] = None): dev_mode = os.environ.get("DEV_MODE", "").lower() in ("true", "1", "yes") self.cache_ttl: int = 0 if dev_mode else 5 # seconds + # Tailscale status cache (separate from config cache, longer TTL) + self._tailscale_hostname: Optional[str] = None + self._tailscale_cache_timestamp: float = 0 + self._tailscale_cache_ttl: int = 30 # 30 seconds - container status doesn't change often + def clear_cache(self) -> None: """Clear the configuration cache, forcing reload on next access.""" self._cache = None self._cache_timestamp = 0 + self._tailscale_hostname = None + self._tailscale_cache_timestamp = 0 logger.info("OmegaConfSettings cache cleared") + def get_tailscale_hostname(self) -> Optional[str]: + """ + Get the Tailscale hostname from TailscaleManager with caching. + + Returns the full DNS name (e.g., "orange.spangled-kettle.ts.net") if + Tailscale is authenticated, None otherwise. + + Uses a 30-second cache to avoid repeated Docker container queries. + """ + # Check cache first + if self._tailscale_hostname is not None: + if time.time() - self._tailscale_cache_timestamp < self._tailscale_cache_ttl: + return self._tailscale_hostname + + # Query TailscaleManager for live status + try: + from src.services.tailscale_manager import get_tailscale_manager + ts_manager = get_tailscale_manager() + ts_status = ts_manager.get_container_status() + + if ts_status.authenticated and ts_status.hostname: + self._tailscale_hostname = ts_status.hostname + self._tailscale_cache_timestamp = time.time() + return self._tailscale_hostname + else: + # Not authenticated - cache the None result too + self._tailscale_hostname = None + self._tailscale_cache_timestamp = time.time() + return None + + except Exception as e: + logger.debug(f"Could not get Tailscale status: {e}") + return None + def _load_yaml_if_exists(self, path: Path) -> Optional[DictConfig]: """Load a YAML file if it exists, return None otherwise.""" if path.exists(): @@ -660,34 +701,41 @@ async def get_suggestions_for_env_var( seen_paths = set() config = await self.get_config_as_dict() - # Determine which sections to search based on env var type - setting_type = infer_setting_type(env_var_name) - sections = SETTING_SECTIONS.get(setting_type, ['api_keys', 'security']) - - # Search config sections - for section in sections: - section_data = config.get(section, {}) - if not isinstance(section_data, dict): - continue - - for key, value in section_data.items(): - if value is None or isinstance(value, dict): - continue + # Determine expected type from env var name + expected_type = infer_setting_type(env_var_name) - path = f"{section}.{key}" + # Helper to recursively collect settings from nested dicts + def collect_settings(data: dict, prefix: str): + for key, value in data.items(): + path = f"{prefix}.{key}" if prefix else key if path in seen_paths: continue - seen_paths.add(path) - - str_value = str(value) if value is not None else "" - has_value = bool(str_value.strip()) - suggestions.append(SettingSuggestion( - path=path, - label=key.replace("_", " ").title(), - has_value=has_value, - value=mask_secret_value(str_value, path) if has_value else None, - )) + if isinstance(value, dict): + # Recursively search nested dicts + collect_settings(value, path) + elif value is not None: + # Filter by type - only show settings that match expected type + # e.g., API_KEY should only show secret-type settings + setting_type = infer_setting_type(key) + if expected_type == 'secret' and setting_type != 'secret': + continue + if expected_type == 'url' and setting_type != 'url': + continue + # 'string' type matches anything + + seen_paths.add(path) + str_value = str(value) if value is not None else "" + has_value = bool(str_value.strip()) + suggestions.append(SettingSuggestion( + path=path, + label=key.replace("_", " ").title(), + has_value=has_value, + value=mask_secret_value(str_value, path) if has_value else None, + )) + + # Search ALL sections in config dynamically + collect_settings(config, "") # Add provider-specific mappings if registry provided if provider_registry and capabilities: diff --git a/ushadow/backend/src/config/yaml_parser.py b/ushadow/backend/src/config/yaml_parser.py index 0cab6df9..b61a8f68 100644 --- a/ushadow/backend/src/config/yaml_parser.py +++ b/ushadow/backend/src/config/yaml_parser.py @@ -209,6 +209,7 @@ class ComposeService: namespace: Optional[str] = None # Docker Compose project name / K8s namespace infra_services: List[str] = field(default_factory=list) # Infra services to start first route_path: Optional[str] = None # Tailscale Serve route path (e.g., "/chronicle") + wizard: Optional[str] = None # Setup wizard ID @property def required_env_vars(self) -> List[ComposeEnvVar]: @@ -361,6 +362,7 @@ def _parse_service( provides = service_meta.get("provides") # Capability this service implements display_name = service_meta.get("display_name") description = service_meta.get("description") + wizard = service_meta.get("wizard") # Setup wizard ID # These are at top level of x-ushadow, shared by all services in file namespace = x_ushadow.get("namespace") infra_services = x_ushadow.get("infra_services", []) @@ -385,6 +387,7 @@ def _parse_service( namespace=namespace, infra_services=infra_services, route_path=route_path, + wizard=wizard, ) def _resolve_image(self, image: Optional[str]) -> Optional[str]: diff --git a/ushadow/backend/src/models/provider.py b/ushadow/backend/src/models/provider.py index 287b31e4..28e1aeab 100644 --- a/ushadow/backend/src/models/provider.py +++ b/ushadow/backend/src/models/provider.py @@ -79,7 +79,7 @@ class Provider(BaseModel): id: str = Field(..., description="Provider identifier (e.g., 'openai', 'ollama')") name: str = Field(..., description="Display name") capability: str = Field(..., description="Which capability this implements") - mode: Literal["cloud", "local"] = Field(..., description="Deployment mode") + mode: Literal["cloud", "local", "client", "relay", "webhook", "custom", "upload", "remote"] = Field(..., description="Deployment mode") description: Optional[str] = None # Environment variable mappings (capability keys → env vars) diff --git a/ushadow/backend/src/routers/audio_provider.py b/ushadow/backend/src/routers/audio_provider.py new file mode 100644 index 00000000..30066b9d --- /dev/null +++ b/ushadow/backend/src/routers/audio_provider.py @@ -0,0 +1,398 @@ +""" +Audio Consumer Router - Exposes audio consumer provider configuration + +Allows mobile apps and audio sources to discover where to send audio. +Audio consumers are services that RECEIVE and PROCESS audio (Chronicle, Mycelia, etc.) +""" +from fastapi import APIRouter, HTTPException, Depends +from pydantic import BaseModel +from typing import Dict, List, Optional, Any +import yaml +import os + +from ..services.auth import get_current_user +from ..config.omegaconf_settings import get_settings_store + +router = APIRouter(prefix="/api/providers/audio_consumer", tags=["providers"]) + + +class AudioConsumerConfig(BaseModel): + """Audio consumer provider configuration""" + provider_id: str + name: str + websocket_url: str + protocol: str + format: str + mode: Optional[str] = None + destinations: Optional[List[Dict[str, str]]] = None + + +class AudioConsumerResponse(BaseModel): + """Response with full audio consumer details""" + capability: str = "audio_consumer" + selected_provider: str + config: AudioConsumerConfig + available_providers: List[str] + + +class SetProviderRequest(BaseModel): + """Request to change active provider""" + provider_id: str + + +def load_audio_consumers() -> Dict[str, Any]: + """Load audio consumer providers from config file""" + config_path = os.path.join( + os.path.dirname(__file__), + "../../..", # Go up to project root + "config/providers/audio_consumer.yaml" + ) + + if not os.path.exists(config_path): + raise HTTPException( + status_code=500, + detail="Audio consumer provider config not found" + ) + + with open(config_path, 'r') as f: + data = yaml.safe_load(f) + + return data + + +async def get_selected_consumer_id() -> str: + """Get the currently selected audio consumer ID""" + config = get_settings_store() + provider_id = await config.get("selected_providers.audio_consumer") + + if not provider_id: + # Default to chronicle if not set + provider_id = "chronicle" + + return provider_id + + +def find_provider_by_id(providers_data: Dict[str, Any], provider_id: str) -> Optional[Dict[str, Any]]: + """Find provider configuration by ID""" + providers = providers_data.get("providers", []) + for provider in providers: + if provider.get("id") == provider_id: + return provider + return None + + +def build_consumer_config(provider: Dict[str, Any]) -> AudioConsumerConfig: + """Build AudioConsumerConfig from provider definition""" + provider_id = provider["id"] + name = provider["name"] + + # Extract credentials + credentials = provider.get("credentials", {}) + websocket_url = credentials.get("websocket_url", {}).get("default", "") + protocol = credentials.get("protocol", {}).get("value", "wyoming") + format_val = credentials.get("format", {}).get("value", "pcm_s16le_16khz_mono") + + # Extract config + config = provider.get("config", {}) + mode = None + destinations = None + + # Get mode from config (recording_mode, processing_mode, etc.) + for key in ["recording_mode", "processing_mode"]: + if key in config: + mode = config[key].get("default") + break + + # Get destinations for relay provider + if "destinations" in config: + dest_str = config["destinations"].get("default", "[]") + try: + import json + destinations = json.loads(dest_str) + except: + destinations = None + + return AudioConsumerConfig( + provider_id=provider_id, + name=name, + websocket_url=websocket_url, + protocol=protocol, + format=format_val, + mode=mode, + destinations=destinations, + ) + + +@router.get("/active", response_model=AudioConsumerResponse) +async def get_active_consumer(user = Depends(get_current_user)): + """ + Get the currently active audio consumer configuration. + + Returns the selected consumer with connection details that mobile apps + and audio sources can use to send audio streams. + """ + try: + # Load consumers config + providers_data = load_audio_consumers() + + # Get selected consumer ID + selected_id = await get_selected_consumer_id() + + # Find provider config + provider = find_provider_by_id(providers_data, selected_id) + if not provider: + raise HTTPException( + status_code=404, + detail=f"Provider '{selected_id}' not found" + ) + + # Build config + config = build_consumer_config(provider) + + # Get available consumers + available = [p["id"] for p in providers_data.get("providers", [])] + + return AudioConsumerResponse( + selected_provider=selected_id, + config=config, + available_providers=available, + ) + + except Exception as e: + raise HTTPException(status_code=500, detail=str(e)) + + +@router.get("/available") +async def get_available_consumers(user = Depends(get_current_user)): + """ + Get list of available audio consumer providers. + + Returns consumer IDs and basic info for all configured consumers. + """ + try: + providers_data = load_audio_consumers() + providers = providers_data.get("providers", []) + + return { + "capability": "audio_consumer", + "providers": [ + { + "id": p["id"], + "name": p["name"], + "description": p["description"], + "mode": p.get("mode", "unknown"), + } + for p in providers + ] + } + + except Exception as e: + raise HTTPException(status_code=500, detail=str(e)) + + +@router.put("/active") +async def set_active_consumer( + request: SetProviderRequest, + user = Depends(get_current_user) +): + """ + Set the active audio consumer. + + Requires admin permission. Updates the selected consumer in configuration. + """ + try: + # TODO: Add admin permission check + # if not user.is_admin: + # raise HTTPException(status_code=403, detail="Admin permission required") + + # Load consumers to validate + providers_data = load_audio_consumers() + provider = find_provider_by_id(providers_data, request.provider_id) + + if not provider: + raise HTTPException( + status_code=404, + detail=f"Consumer '{request.provider_id}' not found" + ) + + # Update selected consumer + config = get_settings_store() + await config.set("selected_providers.audio_consumer", request.provider_id) + + return { + "success": True, + "selected_provider": request.provider_id, + "message": f"Audio consumer set to '{provider['name']}'" + } + + except HTTPException: + raise + except Exception as e: + raise HTTPException(status_code=500, detail=str(e)) + + +@router.get("/health") +async def check_consumer_health(user = Depends(get_current_user)): + """ + Check health of the active audio consumer. + + Attempts to connect to the consumer's WebSocket endpoint. + """ + # TODO: Implement WebSocket health check + return { + "status": "ok", + "message": "Health check not yet implemented" + } + + +class AudioDestination(BaseModel): + """A wired audio destination""" + consumer_id: str + consumer_name: str + websocket_url: str # External URL (for direct connection) or internal URL (for relay) + protocol: str = "wyoming" + format: str = "pcm_s16le_16khz_mono" + + +class WiredDestinationsResponse(BaseModel): + """Response with all wired audio destinations""" + has_destinations: bool + destinations: List[AudioDestination] + # Relay mode: frontend connects to relay_url, backend fans out to destinations + use_relay: bool = False + relay_url: Optional[str] = None # e.g., wss://hostname/ws/audio/relay + + +@router.get("/wired-destinations", response_model=WiredDestinationsResponse) +async def get_wired_audio_destinations(user = Depends(get_current_user)): + """ + Get audio destinations based on wiring configuration. + + Returns WebSocket URLs for all consumers that have audio_input wired to them. + Used by the frontend recording component to know where to send audio. + + If Tailscale is configured, returns wss:// URLs via Tailscale hostname. + Otherwise falls back to ws://localhost for local development. + """ + import yaml + from pathlib import Path + from ..services.service_config_manager import ServiceConfigManager + + destinations = [] + + try: + # Load wiring config from the same location as ServiceConfigManager + config_manager = ServiceConfigManager() + wiring_path = config_manager.wiring_path + if wiring_path.exists(): + with open(wiring_path) as f: + wiring_data = yaml.safe_load(f) or {} + else: + wiring_data = {} + + wiring_list = wiring_data.get("wiring", []) + + # Find all audio_input wiring entries + audio_wiring = [ + w for w in wiring_list + if w.get("source_capability") == "audio_input" or + w.get("target_capability") == "audio_input" + ] + + if not audio_wiring: + # Fall back to checking if any known audio consumers are running + # by looking at the target services + pass + + # Get settings for port resolution and Tailscale hostname + settings = get_settings_store() + + # Get Tailscale hostname from settings store (cached, uses TailscaleManager internally) + tailscale_hostname = settings.get_tailscale_hostname() + + # Get environment name for container naming + env_name = os.environ.get("COMPOSE_PROJECT_NAME", "ushadow").strip() or "ushadow" + + # Determine if we should use relay mode (when Tailscale is enabled) + use_relay = bool(tailscale_hostname) + relay_url = f"wss://{tailscale_hostname}/ws/audio/relay" if use_relay else None + + # For each wired consumer, build the WebSocket URL + # When using relay, we return INTERNAL URLs (backend connects to services) + # When not using relay, we return EXTERNAL URLs (frontend connects directly) + for wire in audio_wiring: + target_id = wire.get("target_config_id", "") + + # Look up the actual service config to get container name + target_config = config_manager.get_service_config(target_id) + + # If container_name is missing, try to discover it from Docker + if target_config and not target_config.container_name: + config_manager.discover_container_info(target_id) + # Refresh the config after discovery + target_config = config_manager.get_service_config(target_id) + + # Extract service type from target_config_id or template + service_type = "" + if target_config: + template_id = target_config.template_id or "" + service_type = template_id.lower() + if not service_type: + service_type = target_id.lower() + + # Build WebSocket URL based on the service + ws_url = None + consumer_name = target_config.name if target_config else target_id + + if "chronicle" in service_type: + consumer_name = "Chronicle" + if use_relay: + # Internal URL - use container_name from deployment or fallback + container_name = (target_config.container_name if target_config else None) or f"{env_name}-chronicle-backend" + port = await settings.get("chronicle.port") or os.environ.get("CHRONICLE_PORT", "8000") + ws_url = f"ws://{container_name}:{port}/ws_pcm" + else: + # Direct localhost connection (no Tailscale) + port = await settings.get("chronicle.port") or os.environ.get("CHRONICLE_PORT", "8080") + ws_url = f"ws://localhost:{port}/ws_pcm" + + elif "mycelia" in service_type: + consumer_name = "Mycelia" + if use_relay: + # Internal URL - use discovered container_name + container_name = target_config.container_name if target_config else None + if not container_name: + import logging + logging.getLogger(__name__).warning( + f"Mycelia service {target_id} has no container_name - container may not be running" + ) + continue # Skip this destination + # Internal port is 5173 (host-mapped to 15173) + internal_port = "5173" + ws_url = f"ws://{container_name}:{internal_port}/ws_pcm" + else: + # Direct localhost connection (no Tailscale) - use host-mapped port + port = await settings.get("mycelia.backend_port") or os.environ.get("MYCELIA_BACKEND_PORT", "15173") + ws_url = f"ws://localhost:{port}/ws_pcm" + + if ws_url: + destinations.append(AudioDestination( + consumer_id=target_id, + consumer_name=consumer_name, + websocket_url=ws_url, + )) + + return WiredDestinationsResponse( + has_destinations=len(destinations) > 0, + destinations=destinations, + use_relay=use_relay, + relay_url=relay_url, + ) + + except Exception as e: + import logging + logging.getLogger(__name__).error(f"Error getting wired destinations: {e}") + return WiredDestinationsResponse( + has_destinations=False, + destinations=[] + ) diff --git a/ushadow/backend/src/routers/audio_relay.py b/ushadow/backend/src/routers/audio_relay.py new file mode 100644 index 00000000..3fba44bf --- /dev/null +++ b/ushadow/backend/src/routers/audio_relay.py @@ -0,0 +1,287 @@ +""" +Audio Relay Router - WebSocket relay to multiple destinations + +Accepts Wyoming protocol audio from mobile app and forwards to: +- Chronicle (/chronicle/ws_pcm) +- Mycelia (/mycelia/ws_pcm) +- Any other configured endpoints + +Mobile connects once to /ws/audio/relay, server handles fanout. +""" +from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Depends, HTTPException +from typing import List, Dict, Optional +import asyncio +import json +import logging +from datetime import datetime + +from ..services.auth import get_current_user + +router = APIRouter(prefix="/ws/audio", tags=["audio"]) +logger = logging.getLogger(__name__) + + +class AudioRelayConnection: + """Manages a single relay destination connection""" + + def __init__(self, name: str, url: str, token: str): + self.name = name + self.url = url + self.token = token + self.ws: Optional[WebSocket] = None + self.connected = False + self.error_count = 0 + self.max_errors = 5 + + async def connect(self): + """Connect to the destination WebSocket""" + try: + import websockets + + # Add token to URL + url_with_token = f"{self.url}?token={self.token}" + logger.info(f"[AudioRelay:{self.name}] Connecting to {self.url}") + + self.ws = await websockets.connect(url_with_token) + self.connected = True + self.error_count = 0 + logger.info(f"[AudioRelay:{self.name}] Connected") + + except Exception as e: + logger.error(f"[AudioRelay:{self.name}] Connection failed: {e}") + self.connected = False + raise + + async def send_text(self, message: str): + """Send text message to destination""" + if not self.connected or not self.ws: + return + + try: + await self.ws.send(message) + except Exception as e: + self.error_count += 1 + logger.error(f"[AudioRelay:{self.name}] Send text error ({self.error_count}/{self.max_errors}): {e}") + + if self.error_count >= self.max_errors: + logger.warning(f"[AudioRelay:{self.name}] Too many errors, disconnecting") + await self.disconnect() + + async def send_binary(self, data: bytes): + """Send binary data to destination""" + if not self.connected or not self.ws: + return + + try: + await self.ws.send(data) + except Exception as e: + self.error_count += 1 + logger.error(f"[AudioRelay:{self.name}] Send binary error ({self.error_count}/{self.max_errors}): {e}") + + if self.error_count >= self.max_errors: + logger.warning(f"[AudioRelay:{self.name}] Too many errors, disconnecting") + await self.disconnect() + + async def disconnect(self): + """Disconnect from destination""" + if self.ws: + try: + await self.ws.close() + except: + pass + self.connected = False + self.ws = None + logger.info(f"[AudioRelay:{self.name}] Disconnected") + + +class AudioRelaySession: + """Manages a relay session with multiple destinations""" + + def __init__(self, destinations: List[Dict[str, str]], token: str): + self.destinations: List[AudioRelayConnection] = [ + AudioRelayConnection( + name=dest["name"], + url=dest["url"], + token=token + ) + for dest in destinations + ] + self.bytes_relayed = 0 + self.chunks_relayed = 0 + + async def connect_all(self): + """Connect to all destinations""" + results = await asyncio.gather( + *[dest.connect() for dest in self.destinations], + return_exceptions=True + ) + + # Log connection results + for dest, result in zip(self.destinations, results): + if isinstance(result, Exception): + logger.error(f"[AudioRelay:{dest.name}] Failed to connect: {result}") + + # Return number of successful connections + connected = sum(1 for dest in self.destinations if dest.connected) + logger.info(f"[AudioRelay] Connected to {connected}/{len(self.destinations)} destinations") + return connected + + async def relay_text(self, message: str): + """Relay text message to all connected destinations""" + await asyncio.gather( + *[dest.send_text(message) for dest in self.destinations if dest.connected], + return_exceptions=True + ) + + async def relay_binary(self, data: bytes): + """Relay binary data to all connected destinations""" + self.bytes_relayed += len(data) + self.chunks_relayed += 1 + + if self.chunks_relayed % 50 == 0: + logger.info(f"[AudioRelay] Relayed {self.chunks_relayed} chunks, {self.bytes_relayed} bytes") + + await asyncio.gather( + *[dest.send_binary(data) for dest in self.destinations if dest.connected], + return_exceptions=True + ) + + async def disconnect_all(self): + """Disconnect from all destinations""" + await asyncio.gather( + *[dest.disconnect() for dest in self.destinations], + return_exceptions=True + ) + logger.info(f"[AudioRelay] Session ended. Total: {self.chunks_relayed} chunks, {self.bytes_relayed} bytes") + + def get_status(self) -> Dict: + """Get current relay status""" + return { + "destinations": [ + { + "name": dest.name, + "connected": dest.connected, + "errors": dest.error_count, + } + for dest in self.destinations + ], + "bytes_relayed": self.bytes_relayed, + "chunks_relayed": self.chunks_relayed, + } + + +@router.websocket("/relay") +async def audio_relay_websocket( + websocket: WebSocket, + # TODO: Implement your auth - this is a placeholder + # user = Depends(get_current_user) +): + """ + Audio relay WebSocket endpoint. + + Query parameters: + - destinations: JSON array of {"name": "chronicle", "url": "ws://host/chronicle/ws_pcm"} + - token: JWT token for authenticating to destinations + + Example: + ws://localhost:8000/ws/audio/relay?destinations=[{"name":"chronicle","url":"ws://localhost:5001/chronicle/ws_pcm"},{"name":"mycelia","url":"ws://localhost:5173/ws_pcm"}]&token=YOUR_JWT + """ + await websocket.accept() + logger.info("[AudioRelay] Client connected") + + # Parse destinations from query params + try: + destinations_param = websocket.query_params.get("destinations") + token = websocket.query_params.get("token") + + if not destinations_param or not token: + await websocket.close(code=1008, reason="Missing destinations or token parameter") + return + + destinations = json.loads(destinations_param) + if not isinstance(destinations, list) or len(destinations) == 0: + await websocket.close(code=1008, reason="destinations must be a non-empty array") + return + + logger.info(f"[AudioRelay] Destinations: {[d['name'] for d in destinations]}") + + except json.JSONDecodeError as e: + await websocket.close(code=1008, reason=f"Invalid destinations JSON: {e}") + return + except Exception as e: + await websocket.close(code=1011, reason=f"Error parsing parameters: {e}") + return + + # Create relay session + session = AudioRelaySession(destinations, token) + + try: + # Connect to all destinations + connected_count = await session.connect_all() + + if connected_count == 0: + await websocket.send_json({ + "type": "error", + "message": "Failed to connect to any destinations" + }) + await websocket.close(code=1011, reason="No destinations available") + return + + # Send status to client + await websocket.send_json({ + "type": "relay_status", + "data": session.get_status() + }) + + # Relay loop + while True: + # Receive from mobile client + try: + message = await websocket.receive() + except WebSocketDisconnect: + logger.info("[AudioRelay] Client disconnected") + break + + # Relay text messages (Wyoming protocol headers) + if "text" in message: + text_data = message["text"] + logger.debug(f"[AudioRelay] Relaying text: {text_data[:100]}") + await session.relay_text(text_data) + + # Relay binary messages (audio chunks) + elif "bytes" in message: + binary_data = message["bytes"] + await session.relay_binary(binary_data) + + except Exception as e: + logger.error(f"[AudioRelay] Error: {e}", exc_info=True) + try: + await websocket.send_json({ + "type": "error", + "message": str(e) + }) + except: + pass + + finally: + # Cleanup + await session.disconnect_all() + try: + await websocket.close() + except: + pass + + +@router.get("/relay/status") +async def relay_status(): + """Get relay endpoint information""" + return { + "endpoint": "/ws/audio/relay", + "protocol": "Wyoming", + "description": "Multi-destination audio relay", + "parameters": { + "destinations": "JSON array of destination configs", + "token": "JWT token for destination authentication" + }, + "example_url": 'ws://localhost:8000/ws/audio/relay?destinations=[{"name":"chronicle","url":"ws://host/chronicle/ws_pcm"}]&token=JWT' + } diff --git a/ushadow/backend/src/routers/services.py b/ushadow/backend/src/routers/services.py index 611380ee..5c68517c 100644 --- a/ushadow/backend/src/routers/services.py +++ b/ushadow/backend/src/routers/services.py @@ -854,6 +854,72 @@ async def uninstall_service( return result +@router.post("/mycelia/generate-token") +async def generate_mycelia_token( + current_user: User = Depends(get_current_user) +) -> Dict[str, str]: + """ + Generate Mycelia authentication token by running the token-create command. + + Returns: + Dictionary with 'token' and 'client_id' fields + """ + import subprocess + import re + from pathlib import Path + from src.services.compose_registry import COMPOSE_DIR + + try: + # Run the docker compose command to generate token + compose_file = COMPOSE_DIR / "mycelia-compose.yml" + if not compose_file.exists(): + raise HTTPException(status_code=500, detail=f"Mycelia compose file not found at {compose_file}") + + result = subprocess.run( + [ + "docker", "compose", "-f", str(compose_file), + "run", "--rm", "mycelia-backend", + "deno", "run", "-A", "server.ts", "token-create" + ], + capture_output=True, + text=True, + timeout=30 + ) + + if result.returncode != 0: + logger.error(f"Failed to generate Mycelia token: {result.stderr}") + raise HTTPException( + status_code=500, + detail=f"Failed to generate token: {result.stderr}" + ) + + # Parse output to extract token and client_id + # Expected format: + # MYCELIA_TOKEN=mycelia_... + # MYCELIA_CLIENT_ID=... + output = result.stdout + token_match = re.search(r'MYCELIA_TOKEN=(\S+)', output) + client_id_match = re.search(r'MYCELIA_CLIENT_ID=(\S+)', output) + + if not token_match or not client_id_match: + logger.error(f"Failed to parse token from output: {output}") + raise HTTPException( + status_code=500, + detail="Failed to parse token from output" + ) + + return { + "token": token_match.group(1), + "client_id": client_id_match.group(1) + } + + except subprocess.TimeoutExpired: + raise HTTPException(status_code=500, detail="Token generation timed out") + except Exception as e: + logger.error(f"Error generating Mycelia token: {e}") + raise HTTPException(status_code=500, detail=str(e)) + + @router.post("/register", response_model=ActionResponse) async def register_dynamic_service( request: RegisterServiceRequest, diff --git a/ushadow/backend/src/routers/sse.py b/ushadow/backend/src/routers/sse.py new file mode 100644 index 00000000..ee6cab83 --- /dev/null +++ b/ushadow/backend/src/routers/sse.py @@ -0,0 +1,169 @@ +""" +SSE Router - Server-Sent Events endpoints for real-time streaming. + +Provides SSE endpoints for operations that benefit from real-time progress: +- Docker image pulls +- Service deployments +- Long-running tasks + +All endpoints support auth via query param (?token=...) since EventSource +doesn't support custom headers. +""" + +import logging +from typing import Optional + +from fastapi import APIRouter, HTTPException, Request + +from src.utils.sse_bridge import SSEBridge, create_sse_response +from src.services.auth import get_user_from_token + +logger = logging.getLogger(__name__) +router = APIRouter() + + +async def validate_sse_auth(request: Request, token: Optional[str] = None): + """ + Validate authentication for SSE endpoints. + + Checks Authorization header first, then falls back to query param. + This is needed because EventSource doesn't support custom headers. + """ + # Check header first + auth_header = request.headers.get("authorization", "") + if auth_header.startswith("Bearer "): + token = auth_header[7:] + + if not token: + raise HTTPException(status_code=401, detail="Authentication required. Pass ?token=your_jwt") + + try: + user = await get_user_from_token(token) + if not user: + raise HTTPException(status_code=401, detail="Invalid token") + return user + except HTTPException: + raise + except Exception as e: + logger.warning(f"Token validation failed: {e}") + raise HTTPException(status_code=401, detail="Invalid token") + + +@router.get("/docker/pull/{service_name}") +async def pull_service_image( + service_name: str, + request: Request, + token: Optional[str] = None, +): + """ + Pull Docker image for a service with streaming progress. + + Auth: Pass token via query param since EventSource doesn't support headers. + GET /api/sse/docker/pull/faster-whisper?token=your_jwt + + Events: + {"status": "Pulling", "id": "abc123", "progress": "[=====> ] 50%"} + {"status": "Downloading", "id": "def456", "progressDetail": {"current": 1234, "total": 5678}} + {"complete": true, "success": true, "message": "Pull complete"} + + Usage (JavaScript): + const token = localStorage.getItem('token'); + const es = new EventSource(`/api/sse/docker/pull/faster-whisper?token=${token}`); + es.onmessage = (e) => { + const data = JSON.parse(e.data); + console.log(data.status, data.progress); + if (data.complete) es.close(); + }; + """ + await validate_sse_auth(request, token) + + from src.services.docker_manager import get_docker_manager + from src.services.compose_registry import get_compose_registry + + docker_mgr = get_docker_manager() + compose_registry = get_compose_registry() + + # Find service + service = compose_registry.get_service_by_name(service_name) + if not service: + raise HTTPException(status_code=404, detail=f"Service '{service_name}' not found") + + if not service.image: + raise HTTPException(status_code=400, detail=f"Service '{service_name}' has no image defined") + + # Parse image:tag + image = service.image + if ":" in image: + image_name, tag = image.rsplit(":", 1) + else: + image_name, tag = image, "latest" + + # Create bridge and pull operation + bridge = SSEBridge() + + def pull_operation(): + """Run Docker pull with progress callbacks.""" + def on_progress(event: dict): + bridge.send(event) + + success, message = docker_mgr.pull_image_with_progress( + image_name, tag, callback=on_progress + ) + bridge.complete(success=success, message=message) + + return create_sse_response(bridge, pull_operation) + + +@router.get("/docker/logs/{service_name}") +async def stream_service_logs( + service_name: str, + request: Request, + token: Optional[str] = None, + tail: int = 50, +): + """ + Stream Docker container logs in real-time. + + Args: + service_name: Name of the service + tail: Number of historical lines to include (default 50) + token: JWT token for auth + + Events: + {"log": "2024-01-20 10:00:00 INFO Starting...", "stream": "stdout"} + {"log": "2024-01-20 10:00:01 ERROR Failed!", "stream": "stderr"} + + Usage: + const es = new EventSource(`/api/sse/docker/logs/chronicle-backend?token=${token}&tail=100`); + """ + await validate_sse_auth(request, token) + + from src.services.docker_manager import get_docker_manager + import docker + + docker_mgr = get_docker_manager() + + if not docker_mgr.is_available(): + raise HTTPException(status_code=503, detail="Docker not available") + + # Get container + container_name = docker_mgr._get_container_name(service_name) + + try: + container = docker_mgr._client.containers.get(container_name) + except docker.errors.NotFound: + raise HTTPException(status_code=404, detail=f"Container '{service_name}' not found") + + bridge = SSEBridge() + + def stream_logs(): + """Stream logs from container.""" + try: + for line in container.logs(stream=True, follow=True, tail=tail): + log_line = line.decode('utf-8', errors='replace').rstrip() + if log_line: + bridge.send({"log": log_line}) + except Exception as e: + bridge.error(str(e)) + + return create_sse_response(bridge, stream_logs) diff --git a/ushadow/backend/src/services/auth.py b/ushadow/backend/src/services/auth.py index b14a815c..3decaa0d 100644 --- a/ushadow/backend/src/services/auth.py +++ b/ushadow/backend/src/services/auth.py @@ -299,6 +299,12 @@ def generate_jwt_for_service( "aud": audiences, # Audiences - services that can use this token "exp": datetime.utcnow() + timedelta(seconds=JWT_LIFETIME_SECONDS), "iat": datetime.utcnow(), + # Mycelia-compatible fields for resource authorization + "principal": user_email, # Identity for access logging + "policies": [ + # Grant full access - fine-grained policies can be added later + {"resource": "**", "action": "*", "effect": "allow"} + ], } return jwt.encode(payload, SECRET_KEY, algorithm=ALGORITHM) diff --git a/ushadow/backend/src/services/capability_resolver.py b/ushadow/backend/src/services/capability_resolver.py index ccfafd5f..f3d4e491 100644 --- a/ushadow/backend/src/services/capability_resolver.py +++ b/ushadow/backend/src/services/capability_resolver.py @@ -641,7 +641,7 @@ async def validate_service(self, service_id: str) -> Dict[str, Any]: capability = use['capability'] required = use.get('required', True) - provider = await self._get_selected_provider(capability) + provider, _ = await self._get_selected_provider(capability) if not provider: if required: missing_caps.append({ @@ -715,7 +715,7 @@ async def get_setup_requirements(self, service_ids: List[str]) -> Dict[str, Any] if capability in seen_capabilities: continue - provider = await self._get_selected_provider(capability) + provider, _ = await self._get_selected_provider(capability) if not provider: if required: seen_capabilities[capability] = { diff --git a/ushadow/backend/src/services/compose_registry.py b/ushadow/backend/src/services/compose_registry.py index 67c5d8f1..50f936ff 100644 --- a/ushadow/backend/src/services/compose_registry.py +++ b/ushadow/backend/src/services/compose_registry.py @@ -121,6 +121,7 @@ class DiscoveredService: namespace: Optional[str] = None # Docker Compose project / K8s namespace infra_services: List[str] = field(default_factory=list) # Infra services to start first route_path: Optional[str] = None # Tailscale Serve route path (e.g., "/chronicle") + wizard: Optional[str] = None # Setup wizard ID from x-ushadow # Environment variables required_env_vars: List[ComposeEnvVar] = field(default_factory=list) @@ -279,6 +280,7 @@ def _load_compose_file(self, filepath: Path) -> None: namespace=service.namespace, infra_services=service.infra_services, route_path=service.route_path, + wizard=service.wizard, required_env_vars=service.required_env_vars, optional_env_vars=service.optional_env_vars, ) diff --git a/ushadow/backend/src/services/docker_manager.py b/ushadow/backend/src/services/docker_manager.py index 563aac10..6eaf408f 100644 --- a/ushadow/backend/src/services/docker_manager.py +++ b/ushadow/backend/src/services/docker_manager.py @@ -935,12 +935,27 @@ async def _build_env_vars_from_compose_config( resolved = {} + # Get provider registry for suggestions + from src.services.provider_registry import get_provider_registry + provider_registry = get_provider_registry() + for env_var in service.all_env_vars: config = saved_config.get(env_var.name, {}) source = config.get("source", "default") setting_path = config.get("setting_path") literal_value = config.get("value") + # Auto-map if no saved config and a matching suggestion with value exists + # This ensures auto-mapping works without requiring manual save + if source == "default" and not setting_path: + suggestions = await settings.get_suggestions_for_env_var( + env_var.name, provider_registry, service.requires + ) + auto_match = settings.find_matching_suggestion(env_var.name, suggestions) + if auto_match: + source = "setting" + setting_path = auto_match.path + # Use settings.resolve_env_value as single source of truth # This ensures UI display and container startup use identical resolution value = await settings.resolve_env_value( @@ -998,6 +1013,17 @@ async def _build_env_vars_for_service( container_env = await self._build_env_vars_from_compose_config(service_name) subprocess_env.update(container_env) + # Add WELL_KNOWN_ENV_MAPPINGS to subprocess_env for Docker Compose variable substitution + # This allows compose files to use ${AUTH_SECRET_KEY} which gets substituted by Docker + from src.services.service_orchestrator import WELL_KNOWN_ENV_MAPPINGS + from src.config.omegaconf_settings import get_settings_store + settings_store = get_settings_store() + for env_name, settings_path in WELL_KNOWN_ENV_MAPPINGS.items(): + if env_name not in subprocess_env: # Don't override already-set values + value = await settings_store.get(settings_path) + if value: + subprocess_env[env_name] = str(value) + # Also try CapabilityResolver for any capabilities declared in x-ushadow requires = service_config.get("metadata", {}).get("requires", []) if requires: @@ -1297,7 +1323,7 @@ async def _start_service_via_compose(self, service_name: str, compose_file: str, cwd=str(compose_dir), capture_output=True, text=True, - timeout=60 + timeout=180 # Increased to 3 minutes for services that need building ) if result.returncode == 0: diff --git a/ushadow/backend/src/services/service_orchestrator.py b/ushadow/backend/src/services/service_orchestrator.py index 727ca72a..57639cda 100644 --- a/ushadow/backend/src/services/service_orchestrator.py +++ b/ushadow/backend/src/services/service_orchestrator.py @@ -44,6 +44,7 @@ "AUTH_SECRET_KEY": "security.auth_secret_key", "ADMIN_PASSWORD": "security.admin_password", "USER": "auth.admin_email", # For OpenMemory backend + "MYCELIA_SECRET_KEY": "security.auth_secret_key", # Mycelia JWT signing } @@ -70,6 +71,7 @@ class ServiceSummary: profiles: List[str] = field(default_factory=list) required_env_count: int = 0 optional_env_count: int = 0 + wizard: Optional[str] = None # ID of setup wizard if available def to_dict(self) -> Dict[str, Any]: return { @@ -89,6 +91,7 @@ def to_dict(self) -> Dict[str, Any]: "profiles": self.profiles, "required_env_count": self.required_env_count, "optional_env_count": self.optional_env_count, + "wizard": self.wizard, } @@ -826,6 +829,7 @@ async def _build_service_summary(self, service: DiscoveredService, installed: bo profiles=service.profiles, required_env_count=len(service.required_env_vars), optional_env_count=len(service.optional_env_vars), + wizard=service.wizard, ) async def _check_needs_setup(self, service: DiscoveredService) -> bool: diff --git a/ushadow/backend/src/services/tailscale_manager.py b/ushadow/backend/src/services/tailscale_manager.py index 6c42aa35..a9e8be40 100644 --- a/ushadow/backend/src/services/tailscale_manager.py +++ b/ushadow/backend/src/services/tailscale_manager.py @@ -677,13 +677,13 @@ def configure_base_routes(self, # Backend API routes - include path in target to preserve it # (Tailscale serve strips the --set-path prefix from the request) - backend_routes = ["/api", "/auth"] + backend_routes = ["/api", "/auth", "/ws"] for route in backend_routes: target = f"{backend_base}{route}" if not self.add_serve_route(route, target): success = False - # WebSocket routes - direct to Chronicle for low latency + # WebSocket routes - direct to Chronicle for low latency (legacy/mobile) ws_routes = ["/ws_pcm", "/ws_omi"] for route in ws_routes: target = f"{chronicle_base}{route}" diff --git a/ushadow/backend/src/utils/sse_bridge.py b/ushadow/backend/src/utils/sse_bridge.py new file mode 100644 index 00000000..86145896 --- /dev/null +++ b/ushadow/backend/src/utils/sse_bridge.py @@ -0,0 +1,217 @@ +""" +SSE Bridge - Reusable utility for streaming blocking operations via Server-Sent Events. + +This module provides a bridge between synchronous/blocking operations and async SSE streams, +allowing the frontend to receive real-time progress updates. + +Usage: + from src.utils.sse_bridge import SSEBridge, create_sse_response + + # Create bridge and run blocking operation + bridge = SSEBridge() + + def my_blocking_operation(): + for i in range(10): + bridge.send({"progress": i * 10, "status": "Processing..."}) + time.sleep(1) + bridge.complete(success=True, message="Done!") + + # Return SSE response + return create_sse_response(bridge, my_blocking_operation) +""" + +import asyncio +import json +import logging +from queue import Queue, Empty +from concurrent.futures import ThreadPoolExecutor +from typing import Any, Callable, Dict, Optional, AsyncGenerator + +from fastapi.responses import StreamingResponse + +logger = logging.getLogger(__name__) + +# Shared thread pool for SSE bridge operations +_sse_executor = ThreadPoolExecutor(max_workers=8, thread_name_prefix="sse-bridge") + + +class SSEBridge: + """ + Bridge for streaming events from blocking operations to SSE. + + The blocking operation calls send() to emit events, and the SSE generator + reads from the internal queue to stream them to the client. + """ + + def __init__(self): + self._queue: Queue = Queue() + self._completed = False + + def send(self, event: Dict[str, Any]) -> None: + """ + Send an event to the SSE stream. + + Args: + event: Dictionary to send as JSON. Common keys: + - status: Current operation status + - progress: Progress percentage or description + - id: Item/layer ID (for multi-item operations) + - error: Error message (if applicable) + """ + self._queue.put(event) + + def complete(self, success: bool = True, message: str = "", **extra) -> None: + """ + Signal completion of the operation. + + Args: + success: Whether operation succeeded + message: Completion message + **extra: Additional fields to include in completion event + """ + event = { + "complete": True, + "success": success, + "message": message, + **extra + } + self._queue.put(event) + self._completed = True + + def error(self, message: str, **extra) -> None: + """ + Signal an error and complete the stream. + + Args: + message: Error message + **extra: Additional error details + """ + event = { + "complete": True, + "success": False, + "error": message, + **extra + } + self._queue.put(event) + self._completed = True + + @property + def is_completed(self) -> bool: + return self._completed + + def get_event(self, timeout: float = 0.1) -> Optional[Dict[str, Any]]: + """Get next event from queue (blocking with timeout).""" + try: + return self._queue.get(timeout=timeout) + except Empty: + return None + + def drain(self) -> list[Dict[str, Any]]: + """Get all remaining events from queue.""" + events = [] + while not self._queue.empty(): + try: + events.append(self._queue.get_nowait()) + except Empty: + break + return events + + +async def generate_sse_events( + bridge: SSEBridge, + operation: Callable[[], None], + executor: Optional[ThreadPoolExecutor] = None +) -> AsyncGenerator[str, None]: + """ + Generate SSE events from a blocking operation. + + Args: + bridge: SSEBridge instance for communication + operation: Blocking callable that uses bridge.send() to emit events + executor: Optional thread pool (uses default if not provided) + + Yields: + SSE-formatted event strings + """ + pool = executor or _sse_executor + loop = asyncio.get_event_loop() + + # Start the blocking operation in a thread + future = loop.run_in_executor(pool, operation) + + try: + while True: + # Check for events with async-friendly polling + try: + event = await asyncio.wait_for( + loop.run_in_executor(None, lambda: bridge.get_event(timeout=0.1)), + timeout=0.2 + ) + + if event: + yield f"data: {json.dumps(event)}\n\n" + + if event.get("complete"): + break + + except asyncio.TimeoutError: + # Check if operation finished + if future.done(): + # Drain remaining events + for event in bridge.drain(): + yield f"data: {json.dumps(event)}\n\n" + + # If no completion event was sent, create one + if not bridge.is_completed: + try: + # Check for exceptions + future.result() + yield f"data: {json.dumps({'complete': True, 'success': True})}\n\n" + except Exception as e: + yield f"data: {json.dumps({'complete': True, 'success': False, 'error': str(e)})}\n\n" + break + continue + + except Exception as e: + logger.error(f"SSE generation error: {e}") + yield f"data: {json.dumps({'complete': True, 'success': False, 'error': str(e)})}\n\n" + + +def create_sse_response( + bridge: SSEBridge, + operation: Callable[[], None], + executor: Optional[ThreadPoolExecutor] = None +) -> StreamingResponse: + """ + Create a FastAPI StreamingResponse for SSE. + + Args: + bridge: SSEBridge instance for communication + operation: Blocking callable that uses bridge.send() to emit events + executor: Optional thread pool + + Returns: + StreamingResponse configured for SSE + + Example: + @router.get("/progress") + async def get_progress(): + bridge = SSEBridge() + + def my_operation(): + for i in range(100): + bridge.send({"progress": i}) + time.sleep(0.1) + bridge.complete(success=True) + + return create_sse_response(bridge, my_operation) + """ + return StreamingResponse( + generate_sse_events(bridge, operation, executor), + media_type="text/event-stream", + headers={ + "Cache-Control": "no-cache", + "Connection": "keep-alive", + "X-Accel-Buffering": "no", # Disable nginx buffering + } + ) diff --git a/ushadow/frontend/src/App.tsx b/ushadow/frontend/src/App.tsx index c7902358..cd3f3032 100644 --- a/ushadow/frontend/src/App.tsx +++ b/ushadow/frontend/src/App.tsx @@ -51,6 +51,7 @@ import { LocalServicesWizard, MobileAppWizard, SpeakerRecognitionWizard, + MyceliaWizard, } from './wizards' import KubernetesClustersPage from './pages/KubernetesClustersPage' import ColorSystemPreview from './components/ColorSystemPreview' @@ -98,6 +99,7 @@ function AppContent() { } /> } /> } /> + } /> } /> } /> } /> diff --git a/ushadow/frontend/src/components/chronicle/ChronicleRecording.tsx b/ushadow/frontend/src/components/chronicle/ChronicleRecording.tsx index d64cf27e..b98cb4e7 100644 --- a/ushadow/frontend/src/components/chronicle/ChronicleRecording.tsx +++ b/ushadow/frontend/src/components/chronicle/ChronicleRecording.tsx @@ -1,10 +1,10 @@ import { useEffect, useRef } from 'react' import { Mic, MicOff, Loader2, Zap, Archive, AlertCircle, Monitor } from 'lucide-react' -import { ChronicleRecordingReturn, RecordingStep } from '../../hooks/useChronicleRecording' +import { WebRecordingReturn, RecordingStep } from '../../hooks/useWebRecording' interface ChronicleRecordingProps { onAuthRequired?: () => void - recording: ChronicleRecordingReturn + recording: WebRecordingReturn } const getStepText = (step: RecordingStep): string => { @@ -12,7 +12,7 @@ const getStepText = (step: RecordingStep): string => { case 'idle': return 'Ready to Record' case 'mic': return 'Getting Microphone Access...' case 'display': return 'Requesting Tab/Screen Audio...' - case 'websocket': return 'Connecting to Chronicle...' + case 'websocket': return 'Connecting to Audio Services...' case 'audio-start': return 'Initializing Audio Session...' case 'streaming': return 'Starting Audio Stream...' case 'stopping': return 'Stopping Recording...' diff --git a/ushadow/frontend/src/components/layout/Layout.tsx b/ushadow/frontend/src/components/layout/Layout.tsx index 2a33dae0..94638989 100644 --- a/ushadow/frontend/src/components/layout/Layout.tsx +++ b/ushadow/frontend/src/components/layout/Layout.tsx @@ -1,13 +1,14 @@ import { Link, useLocation, Outlet } from 'react-router-dom' import React, { useState, useRef, useEffect } from 'react' import { Layers, MessageSquare, Plug, Bot, Workflow, Server, Settings, LogOut, Sun, Moon, Users, Search, Bell, User, ChevronDown, Brain, Home, QrCode } from 'lucide-react' -import { LayoutDashboard, Network, Flag, FlaskConical, Cloud, Mic, MicOff, Loader2, Sparkles } from 'lucide-react' +import { LayoutDashboard, Network, Flag, FlaskConical, Cloud, Mic, MicOff, Loader2, Sparkles, Zap, Archive } from 'lucide-react' import { useAuth } from '../../contexts/AuthContext' import { useTheme } from '../../contexts/ThemeContext' import { useFeatureFlags } from '../../contexts/FeatureFlagsContext' import { useWizard } from '../../contexts/WizardContext' import { useChronicle } from '../../contexts/ChronicleContext' import { useMobileQrCode } from '../../hooks/useQrCode' +import { svcConfigsApi } from '../../services/api' import FeatureFlagsDrawer from './FeatureFlagsDrawer' import { StatusBadge, type BadgeVariant } from '../StatusBadge' import Modal from '../Modal' @@ -35,6 +36,7 @@ export default function Layout() { const [searchQuery, setSearchQuery] = useState('') const [featureFlagsDrawerOpen, setFeatureFlagsDrawerOpen] = useState(false) const userMenuRef = useRef(null) + const [isDesktopMicWired, setIsDesktopMicWired] = useState(false) // QR code hook const { qrData, loading: loadingQrCode, showModal: showQrModal, fetchQrCode, closeModal } = useMobileQrCode() @@ -55,6 +57,28 @@ export default function Layout() { return () => document.removeEventListener('mousedown', handleClickOutside) }, []) + // Check if desktop-mic (or any audio_input provider) is wired + useEffect(() => { + const checkAudioWiring = async () => { + try { + const wiringRes = await svcConfigsApi.getWiring() + // Check if any audio_input provider is wired + const hasAudioWiring = wiringRes.data.some( + (w: any) => w.source_capability === 'audio_input' || + w.source_config_id?.includes('desktop-mic') || + w.source_config_id?.includes('mobile-app') + ) + setIsDesktopMicWired(hasAudioWiring) + } catch (err) { + // Silently fail - user might not be logged in yet + } + } + checkAudioWiring() + // Re-check periodically (every 30 seconds) + const interval = setInterval(checkAudioWiring, 30000) + return () => clearInterval(interval) + }, []) + // Define navigation items with optional feature flag requirements const allNavigationItems: NavigationItem[] = [ // Separator after wizard section @@ -185,36 +209,82 @@ export default function Layout() { {/* Header Actions */}
- {/* Chronicle Record Button - only show when connected */} - {isChronicleConnected && ( - + +
)} - - {recording.isRecording - ? recording.formatDuration(recording.recordingDuration) - : isRecordingProcessing - ? 'Starting...' - : 'Record'} - - + + {/* Record/Stop Button */} + + )} {/* Test Feature Flag Indicator */} diff --git a/ushadow/frontend/src/components/wiring/OutputWiring.tsx b/ushadow/frontend/src/components/wiring/OutputWiring.tsx new file mode 100644 index 00000000..23847fe2 --- /dev/null +++ b/ushadow/frontend/src/components/wiring/OutputWiring.tsx @@ -0,0 +1,570 @@ +/** + * OutputWiring - Components for wiring service outputs to env vars + * + * This module provides: + * - OutputPort: Draggable port on service outputs + * - EnvVarDropTarget: Droppable target for env vars + * - WireOverlay: SVG overlay for drawing connection wires + */ + +import { useState, useRef, useEffect, useCallback } from 'react' +import { createPortal } from 'react-dom' +import { Circle, Link2, X, ExternalLink } from 'lucide-react' + +// ============================================================================= +// Types +// ============================================================================= + +export interface OutputInfo { + instanceId: string + instanceName: string + outputKey: string // "access_url" | "env_vars.XXX" | "capability_values.XXX" + outputLabel: string + value?: string +} + +export interface EnvVarInfo { + instanceId: string + instanceName: string + envVarKey: string + envVarLabel: string + currentValue?: string +} + +export interface OutputWiringConnection { + id: string + source: OutputInfo + target: EnvVarInfo +} + +export interface WirePosition { + sourceId: string + targetId: string + sourcePos: { x: number; y: number } + targetPos: { x: number; y: number } +} + +// ============================================================================= +// OutputPort - Draggable port on service outputs +// ============================================================================= + +interface OutputPortProps { + output: OutputInfo + isConnected: boolean + connectionCount: number + onStartDrag: (output: OutputInfo, startPos: { x: number; y: number }) => void + onEndDrag: () => void + portRef?: (el: HTMLDivElement | null) => void +} + +export function OutputPort({ + output, + isConnected, + connectionCount, + onStartDrag, + onEndDrag, + portRef, +}: OutputPortProps) { + const [isDragging, setIsDragging] = useState(false) + const localRef = useRef(null) + + const handleMouseDown = (e: React.MouseEvent) => { + e.preventDefault() + e.stopPropagation() + setIsDragging(true) + const rect = localRef.current?.getBoundingClientRect() + if (rect) { + onStartDrag(output, { + x: rect.left + rect.width / 2, + y: rect.top + rect.height / 2, + }) + } + } + + useEffect(() => { + if (!isDragging) return + + const handleMouseUp = () => { + setIsDragging(false) + onEndDrag() + } + + window.addEventListener('mouseup', handleMouseUp) + return () => window.removeEventListener('mouseup', handleMouseUp) + }, [isDragging, onEndDrag]) + + const setRefs = (el: HTMLDivElement | null) => { + localRef.current = el + portRef?.(el) + } + + return ( +
+ {/* Output label */} + + {output.outputLabel} + + + {/* Port circle */} +
+ + {/* Connection count badge */} + {connectionCount > 0 && ( + + {connectionCount} + + )} +
+ ) +} + +// ============================================================================= +// EnvVarDropTarget - Droppable target for env vars +// ============================================================================= + +interface EnvVarDropTargetProps { + envVar: EnvVarInfo + isConnected: boolean + connectedFrom?: OutputInfo + isDropTarget: boolean + onDisconnect: () => void + targetRef?: (el: HTMLDivElement | null) => void +} + +export function EnvVarDropTarget({ + envVar, + isConnected, + connectedFrom, + isDropTarget, + onDisconnect, + targetRef, +}: EnvVarDropTargetProps) { + const localRef = useRef(null) + + const setRefs = (el: HTMLDivElement | null) => { + localRef.current = el + targetRef?.(el) + } + + return ( +
+ {/* Target circle */} +
+ + {/* Env var label */} +
+ + {envVar.envVarLabel} + + {isConnected && connectedFrom && ( + + + {connectedFrom.instanceName}.{connectedFrom.outputLabel} + + )} + {!isConnected && isDropTarget && ( + + Drop to connect + + )} +
+ + {/* Disconnect button */} + {isConnected && ( + + )} +
+ ) +} + +// ============================================================================= +// WireOverlay - SVG overlay for drawing connection wires +// ============================================================================= + +interface WireOverlayProps { + wires: WirePosition[] + draggingWire?: { + sourcePos: { x: number; y: number } + currentPos: { x: number; y: number } + } + containerRef: React.RefObject +} + +export function WireOverlay({ wires, draggingWire, containerRef }: WireOverlayProps) { + const [dimensions, setDimensions] = useState({ width: 0, height: 0 }) + const [offset, setOffset] = useState({ x: 0, y: 0 }) + + // Update dimensions on resize + useEffect(() => { + const updateDimensions = () => { + if (containerRef.current) { + const rect = containerRef.current.getBoundingClientRect() + setDimensions({ width: rect.width, height: rect.height }) + setOffset({ x: rect.left, y: rect.top }) + } + } + + updateDimensions() + window.addEventListener('resize', updateDimensions) + window.addEventListener('scroll', updateDimensions) + + // Also update on container size changes + const resizeObserver = new ResizeObserver(updateDimensions) + if (containerRef.current) { + resizeObserver.observe(containerRef.current) + } + + return () => { + window.removeEventListener('resize', updateDimensions) + window.removeEventListener('scroll', updateDimensions) + resizeObserver.disconnect() + } + }, [containerRef]) + + // Convert absolute position to relative position within container + const toRelative = useCallback((pos: { x: number; y: number }) => ({ + x: pos.x - offset.x, + y: pos.y - offset.y, + }), [offset]) + + // Generate a curved path between two points + const generatePath = (start: { x: number; y: number }, end: { x: number; y: number }) => { + const dx = end.x - start.x + const controlPointOffset = Math.min(Math.abs(dx) * 0.5, 100) + + return `M ${start.x} ${start.y} + C ${start.x + controlPointOffset} ${start.y}, + ${end.x - controlPointOffset} ${end.y}, + ${end.x} ${end.y}` + } + + if (dimensions.width === 0 || dimensions.height === 0) return null + + return createPortal( + + + {/* Gradient for connected wires */} + + + + + {/* Gradient for dragging wire */} + + + + + {/* Arrow marker */} + + + + + + {/* Existing connections */} + {wires.map((wire) => { + const startRel = toRelative(wire.sourcePos) + const endRel = toRelative(wire.targetPos) + + return ( + + {/* Shadow/glow effect */} + + {/* Main wire */} + + + ) + })} + + {/* Currently dragging wire */} + {draggingWire && ( + + + {/* Dragging endpoint circle */} + + + )} + , + document.body + ) +} + +// ============================================================================= +// OutputSection - Section showing all outputs for a service +// ============================================================================= + +interface OutputSectionProps { + instanceId: string + instanceName: string + outputs: { + access_url?: string + env_vars?: Record + capability_values?: Record + } + connections: OutputWiringConnection[] + onStartDrag: (output: OutputInfo, startPos: { x: number; y: number }) => void + onEndDrag: () => void + portRefs: Map +} + +export function OutputSection({ + instanceId, + instanceName, + outputs, + connections, + onStartDrag, + onEndDrag, + portRefs, +}: OutputSectionProps) { + // Build list of available outputs + const outputList: OutputInfo[] = [] + + // Add access_url if present + if (outputs.access_url) { + outputList.push({ + instanceId, + instanceName, + outputKey: 'access_url', + outputLabel: 'URL', + value: outputs.access_url, + }) + } + + // Add env_vars if present + if (outputs.env_vars) { + Object.entries(outputs.env_vars).forEach(([key, value]) => { + outputList.push({ + instanceId, + instanceName, + outputKey: `env_vars.${key}`, + outputLabel: key, + value, + }) + }) + } + + // Add capability_values if present + if (outputs.capability_values) { + Object.entries(outputs.capability_values).forEach(([key, value]) => { + outputList.push({ + instanceId, + instanceName, + outputKey: `capability_values.${key}`, + outputLabel: key, + value: typeof value === 'string' ? value : JSON.stringify(value), + }) + }) + } + + if (outputList.length === 0) { + return null + } + + return ( +
+
+ + Outputs +
+
+ {outputList.map((output) => { + const connectionCount = connections.filter( + (c) => c.source.instanceId === instanceId && c.source.outputKey === output.outputKey + ).length + + return ( + 0} + connectionCount={connectionCount} + onStartDrag={onStartDrag} + onEndDrag={onEndDrag} + portRef={(el) => { + if (el) { + portRefs.set(`output::${instanceId}::${output.outputKey}`, el) + } + }} + /> + ) + })} +
+
+ ) +} + +// ============================================================================= +// EnvVarSection - Section showing env vars that can receive wire connections +// ============================================================================= + +interface EnvVarSectionProps { + instanceId: string + instanceName: string + envVars: Array<{ + key: string + label: string + value?: string + required?: boolean + }> + connections: OutputWiringConnection[] + draggingOutput: OutputInfo | null + onDisconnect: (connectionId: string) => void + targetRefs: Map + isDropTarget: (envVarKey: string) => boolean +} + +export function EnvVarSection({ + instanceId, + instanceName, + envVars, + connections, + draggingOutput, + onDisconnect, + targetRefs, + isDropTarget, +}: EnvVarSectionProps) { + if (envVars.length === 0) { + return null + } + + return ( +
+
+ + Env Vars +
+
+ {envVars.map((envVar) => { + const connection = connections.find( + (c) => c.target.instanceId === instanceId && c.target.envVarKey === envVar.key + ) + + return ( + { + if (connection) { + onDisconnect(connection.id) + } + }} + targetRef={(el) => { + if (el) { + targetRefs.set(`envvar::${instanceId}::${envVar.key}`, el) + } + }} + /> + ) + })} +
+
+ ) +} diff --git a/ushadow/frontend/src/components/wiring/WiringBoard.tsx b/ushadow/frontend/src/components/wiring/WiringBoard.tsx index 03ea423b..c0e678be 100644 --- a/ushadow/frontend/src/components/wiring/WiringBoard.tsx +++ b/ushadow/frontend/src/components/wiring/WiringBoard.tsx @@ -6,9 +6,10 @@ * - Right: Consumer instances with capability slots (targets) * * Drag from a provider to a consumer's capability slot to create a connection. + * Also supports output-to-env-var wiring with visual wire connections. */ -import { useState, useCallback, useEffect } from 'react' +import { useState, useCallback, useEffect, useRef } from 'react' import { createPortal } from 'react-dom' import { DndContext, @@ -39,6 +40,8 @@ import { ChevronUp, Plug, Package, + ExternalLink, + Link2, } from 'lucide-react' import { ServiceTemplateCard } from './ServiceTemplateCard' import { ServiceInstanceCard } from './ServiceInstanceCard' @@ -63,6 +66,12 @@ interface ProviderInfo { templateId: string // For templates: own ID; for instances: parent template ID configVars: ConfigVar[] configured?: boolean // Whether all required config fields have values + // Output wiring: available outputs for this provider + outputs?: { + access_url?: string + env_vars?: Record + capability_values?: Record + } } interface ConsumerInfo { @@ -76,6 +85,13 @@ interface ConsumerInfo { description?: string // Service description isTemplate?: boolean // True for templates, false for instances templateId?: string // For templates: own ID; for instances: parent template ID + // Output wiring: env vars that can receive wired values + wirableEnvVars?: Array<{ + key: string + label: string + value?: string + required?: boolean + }> } interface WiringInfo { @@ -86,12 +102,35 @@ interface WiringInfo { target_capability: string } +// Output wiring types +export interface OutputWiringInfo { + id: string + source_instance_id: string + source_output_key: string + target_instance_id: string + target_env_var: string +} + +export interface OutputInfo { + instanceId: string + instanceName: string + outputKey: string + outputLabel: string + value?: string +} + interface DropInfo { provider: ProviderInfo consumerId: string capability: string } +interface OutputDropInfo { + source: OutputInfo + targetInstanceId: string + targetEnvVar: string +} + interface WiringBoardProps { providers: ProviderInfo[] consumers: ConsumerInfo[] @@ -108,6 +147,10 @@ interface WiringBoardProps { onEditConsumer?: (consumerId: string) => void onStartConsumer?: (consumerId: string) => Promise onStopConsumer?: (consumerId: string) => Promise + // Output wiring props + outputWiring?: OutputWiringInfo[] + onOutputWiringCreate?: (dropInfo: OutputDropInfo) => Promise + onOutputWiringDelete?: (wiringId: string) => Promise onDeployConsumer?: (consumerId: string, target: { type: 'local' | 'remote' | 'kubernetes'; id?: string }) => void } @@ -127,6 +170,9 @@ export default function WiringBoard({ onStartConsumer, onDeployConsumer, onStopConsumer, + outputWiring = [], + onOutputWiringCreate, + onOutputWiringDelete, }: WiringBoardProps) { const [activeProvider, setActiveProvider] = useState(null) const [mousePos, setMousePos] = useState({ x: 0, y: 0 }) @@ -167,6 +213,14 @@ export default function WiringBoard({ setSelectingSlot(null) } + // Output wiring state + const [draggingOutput, setDraggingOutput] = useState(null) + const [wireStartPos, setWireStartPos] = useState<{ x: number; y: number } | null>(null) + const [hoveredTarget, setHoveredTarget] = useState(null) + const boardRef = useRef(null) + const outputPortRefs = useRef>(new Map()) + const envVarTargetRefs = useRef>(new Map()) + // Configure sensors for proper drag handling const mouseSensor = useSensor(MouseSensor, { activationConstraint: { @@ -192,6 +246,82 @@ export default function WiringBoard({ return () => window.removeEventListener('mousemove', handleMouseMove) }, [activeProvider]) + // Output wire dragging handlers + const handleOutputDragStart = useCallback((output: OutputInfo, startPos: { x: number; y: number }) => { + setDraggingOutput(output) + setWireStartPos(startPos) + }, []) + + const handleOutputDragEnd = useCallback(async () => { + if (draggingOutput && hoveredTarget && onOutputWiringCreate) { + // Parse target: "envvar::instanceId::envVarKey" + const parts = hoveredTarget.split('::') + if (parts.length === 3 && parts[0] === 'envvar') { + await onOutputWiringCreate({ + source: draggingOutput, + targetInstanceId: parts[1], + targetEnvVar: parts[2], + }) + } + } + setDraggingOutput(null) + setWireStartPos(null) + setHoveredTarget(null) + }, [draggingOutput, hoveredTarget, onOutputWiringCreate]) + + // Track mouse position and detect drop targets during output wire drag + useEffect(() => { + if (!draggingOutput) return + + const handleMouseMove = (e: MouseEvent) => { + setMousePos({ x: e.clientX, y: e.clientY }) + + // Check if hovering over any env var target + const elements = document.elementsFromPoint(e.clientX, e.clientY) + const targetEl = elements.find(el => el.getAttribute('data-target-id')?.startsWith('envvar::')) + if (targetEl) { + setHoveredTarget(targetEl.getAttribute('data-target-id')) + } else { + setHoveredTarget(null) + } + } + + window.addEventListener('mousemove', handleMouseMove) + return () => window.removeEventListener('mousemove', handleMouseMove) + }, [draggingOutput]) + + // Calculate wire positions for existing output wiring connections + const getWirePositions = useCallback(() => { + const positions: Array<{ + sourceId: string + targetId: string + sourcePos: { x: number; y: number } + targetPos: { x: number; y: number } + }> = [] + + outputWiring.forEach((wire) => { + const sourceKey = `output::${wire.source_instance_id}::${wire.source_output_key}` + const targetKey = `envvar::${wire.target_instance_id}::${wire.target_env_var}` + + const sourceEl = outputPortRefs.current.get(sourceKey) + const targetEl = envVarTargetRefs.current.get(targetKey) + + if (sourceEl && targetEl) { + const sourceRect = sourceEl.getBoundingClientRect() + const targetRect = targetEl.getBoundingClientRect() + + positions.push({ + sourceId: sourceKey, + targetId: targetKey, + sourcePos: { x: sourceRect.right, y: sourceRect.top + sourceRect.height / 2 }, + targetPos: { x: targetRect.left, y: targetRect.top + targetRect.height / 2 }, + }) + } + }) + + return positions + }, [outputWiring]) + // Group providers by capability, then by template (templates with their instances nested) const providersByCapability = providers.reduce( (acc, provider) => { @@ -288,7 +418,8 @@ export default function WiringBoard({ collisionDetection={pointerWithin} >
{/* Left Column: Providers */} @@ -357,6 +488,10 @@ export default function WiringBoard({ onStart={onStartProvider ? () => onStartProvider(instance.id, false) : undefined} onStop={onStopProvider ? () => onStopProvider(instance.id, false) : undefined} templateProvider={template} + outputWiring={outputWiring} + onOutputDragStart={handleOutputDragStart} + onOutputDragEnd={handleOutputDragEnd} + outputPortRefs={outputPortRefs.current} /> ) })} @@ -618,6 +753,11 @@ interface DraggableProviderProps { onStart?: () => Promise onStop?: () => Promise templateProvider?: ProviderInfo // Parent template for instances + // Output wiring props + outputWiring?: OutputWiringInfo[] + onOutputDragStart?: (output: OutputInfo, startPos: { x: number; y: number }) => void + onOutputDragEnd?: () => void + outputPortRefs?: Map } function DraggableProvider({ provider, connectionCount, onEdit, onCreateServiceConfig, onDelete, onStart, onStop, templateProvider }: DraggableProviderProps) { @@ -670,163 +810,427 @@ function DraggableProvider({ provider, connectionCount, onEdit, onCreateServiceC
- {/* Draggable header */} -
- -
-
- - {provider.name} - -
- {/* Capability tag */} - - {provider.capability} + {/* Left side: Main content */} +
+ {/* Draggable header */} +
+ +
+
+ + {provider.name} - {isConnected && ( - - {connectionCount} +
+ {/* Capability tag */} + + {provider.capability} - )} + {isConnected && ( + + {connectionCount} + + )} +
-
- {/* Action buttons */} -
- {/* Start/Stop/Setup buttons for local providers */} - {!isCloud && onStart && onStop && ( - <> - {isStarting ? ( - - - - ) : needsSetup && canStart ? ( - - ) : canStart ? ( - - ) : canStop ? ( - - ) : null} - - )} - - {provider.isTemplate && onCreateServiceConfig && ( - - )} - {!provider.isTemplate && onDelete && ( + {/* Action buttons */} +
+ {/* Start/Stop/Setup buttons for local providers */} + {!isCloud && onStart && onStop && ( + <> + {isStarting ? ( + + + + ) : needsSetup && canStart ? ( + + ) : canStart ? ( + + ) : canStop ? ( + + ) : null} + + )} - )} -
- - -
- - {/* Config vars display - show missing required first, then configured */} - {(missingRequiredVars.length > 0 || configuredVars.length > 0) && ( -
-
- {/* Missing required fields - shown first with warning */} - {missingRequiredVars.slice(0, 2).map((v) => ( - handleButtonClick(e, onCreateServiceConfig)} + className="p-1 text-neutral-400 hover:text-primary-600 dark:hover:text-primary-400 hover:bg-neutral-100 dark:hover:bg-neutral-700 rounded" + title="Create new instance" + data-testid={`provider-create-instance-${provider.id}`} > - * - {v.label}:{' '} - Not set - - ))} - {missingRequiredVars.length > 2 && ( - - +{missingRequiredVars.length - 2} required - + + + )} + {!provider.isTemplate && onDelete && ( + )} - {/* Configured fields - color code overrides */} - {configuredVars.slice(0, 3 - Math.min(missingRequiredVars.length, 2)).map((v) => { - // Check if this value is overridden from template - const isOverridden = templateProvider && - templateProvider.configVars.find(tv => tv.key === v.key)?.value !== v.value +
+ + +
- return ( + {/* Config vars display - show missing required first, then configured */} + {(missingRequiredVars.length > 0 || configuredVars.length > 0) && ( +
+
+ {/* Missing required fields - shown first with warning */} + {missingRequiredVars.slice(0, 2).map((v) => ( - {v.required && *} - {v.label}:{' '} - {v.value} + * + {v.label}:{' '} + Not set - ) - })} - {configuredVars.length > (3 - Math.min(missingRequiredVars.length, 2)) && ( - - +{configuredVars.length - (3 - Math.min(missingRequiredVars.length, 2))} more - - )} + ))} + {missingRequiredVars.length > 2 && ( + + +{missingRequiredVars.length - 2} required + + )} + {/* Configured fields - color code overrides */} + {configuredVars.slice(0, 3 - Math.min(missingRequiredVars.length, 2)).map((v) => { + // Check if this value is overridden from template + const isOverridden = templateProvider && + templateProvider.configVars.find(tv => tv.key === v.key)?.value !== v.value + + return ( + + {v.required && *} + {v.label}:{' '} + {v.value} + + ) + })} + {configuredVars.length > (3 - Math.min(missingRequiredVars.length, 2)) && ( + + +{configuredVars.length - (3 - Math.min(missingRequiredVars.length, 2))} more + + )} +
-
+ )} +
+ + {/* Right side: Output ports for wiring */} + {provider.outputs && onOutputDragStart && ( + {})} + outputPortRefs={outputPortRefs} + /> )}
) } +// ============================================================================= +// Output Ports Section - Shows draggable output ports on providers +// ============================================================================= + +interface OutputPortsSectionProps { + provider: ProviderInfo + outputWiring: OutputWiringInfo[] + onOutputDragStart: (output: OutputInfo, startPos: { x: number; y: number }) => void + onOutputDragEnd: () => void + outputPortRefs?: Map +} + +function OutputPortsSection({ + provider, + outputWiring, + onOutputDragStart, + onOutputDragEnd, + outputPortRefs, +}: OutputPortsSectionProps) { + const outputs = provider.outputs + if (!outputs) return null + + // Build list of available outputs + const outputList: Array<{ key: string; label: string; value?: string }> = [] + + if (outputs.access_url) { + outputList.push({ key: 'access_url', label: 'URL', value: outputs.access_url }) + } + + if (outputs.env_vars) { + Object.entries(outputs.env_vars).forEach(([key, value]) => { + outputList.push({ key: `env_vars.${key}`, label: key, value }) + }) + } + + if (outputs.capability_values) { + Object.entries(outputs.capability_values).forEach(([key, value]) => { + outputList.push({ + key: `capability_values.${key}`, + label: key, + value: typeof value === 'string' ? value : JSON.stringify(value), + }) + }) + } + + if (outputList.length === 0) return null + + return ( +
+
+ + Outputs +
+
+ {outputList.map((output) => { + const connectionCount = outputWiring.filter( + (w) => w.source_instance_id === provider.id && w.source_output_key === output.key + ).length + + return ( + { + if (el && outputPortRefs) { + outputPortRefs.set(`output::${provider.id}::${output.key}`, el) + } + }} + /> + ) + })} +
+
+ ) +} + +// ============================================================================= +// Output Ports Side Panel - Shows outputs on the right side of provider cards +// ============================================================================= + +function OutputPortsSidePanel({ + provider, + outputWiring, + onOutputDragStart, + onOutputDragEnd, + outputPortRefs, +}: OutputPortsSectionProps) { + const outputs = provider.outputs + if (!outputs) return null + + // Build list of available outputs + const outputList: Array<{ key: string; label: string; value?: string }> = [] + + if (outputs.access_url) { + outputList.push({ key: 'access_url', label: 'URL', value: outputs.access_url }) + } + + if (outputs.env_vars) { + Object.entries(outputs.env_vars).forEach(([key, value]) => { + outputList.push({ key: `env_vars.${key}`, label: key, value }) + }) + } + + if (outputs.capability_values) { + Object.entries(outputs.capability_values).forEach(([key, value]) => { + outputList.push({ + key: `capability_values.${key}`, + label: key, + value: typeof value === 'string' ? value : JSON.stringify(value), + }) + }) + } + + if (outputList.length === 0) return null + + return ( +
+ {outputList.map((output) => { + const connectionCount = outputWiring.filter( + (w) => w.source_instance_id === provider.id && w.source_output_key === output.key + ).length + + return ( + { + if (el && outputPortRefs) { + outputPortRefs.set(`output::${provider.id}::${output.key}`, el) + } + }} + /> + ) + })} +
+ ) +} + +// ============================================================================= +// Output Port Pill - Individual draggable output port +// ============================================================================= + +interface OutputPortPillProps { + instanceId: string + instanceName: string + outputKey: string + outputLabel: string + value?: string + connectionCount: number + onDragStart: (output: OutputInfo, startPos: { x: number; y: number }) => void + onDragEnd: () => void + portRef?: (el: HTMLDivElement | null) => void +} + +function OutputPortPill({ + instanceId, + instanceName, + outputKey, + outputLabel, + value, + connectionCount, + onDragStart, + onDragEnd, + portRef, +}: OutputPortPillProps) { + const [isDragging, setIsDragging] = useState(false) + const localRef = useRef(null) + + const handleMouseDown = (e: React.MouseEvent) => { + e.preventDefault() + e.stopPropagation() + setIsDragging(true) + const rect = localRef.current?.getBoundingClientRect() + if (rect) { + onDragStart( + { instanceId, instanceName, outputKey, outputLabel, value }, + { x: rect.right, y: rect.top + rect.height / 2 } + ) + } + } + + useEffect(() => { + if (!isDragging) return + + const handleMouseUp = () => { + setIsDragging(false) + onDragEnd() + } + + window.addEventListener('mouseup', handleMouseUp) + return () => window.removeEventListener('mouseup', handleMouseUp) + }, [isDragging, onDragEnd]) + + const setRefs = (el: HTMLDivElement | null) => { + localRef.current = el + portRef?.(el) + } + + const isConnected = connectionCount > 0 + + return ( +
+ {outputLabel} +
+ {connectionCount > 1 && ( + + {connectionCount} + + )} +
+ ) +} + interface ServiceCardProps { consumer: ConsumerInfo @@ -841,6 +1245,13 @@ interface ServiceCardProps { onEdit?: (consumerId: string) => void onStart?: (consumerId: string) => Promise onStop?: (consumerId: string) => Promise + // Output wiring props + outputWiring?: OutputWiringInfo[] + draggingOutput?: OutputInfo | null + hoveredTarget?: string | null + onOutputWiringDelete?: (wiringId: string) => Promise + envVarTargetRefs?: Map + providers?: ProviderInfo[] // To get source instance names onDeploy?: (consumerId: string, target: { type: 'local' | 'remote' | 'kubernetes'; id?: string }) => void } @@ -858,6 +1269,12 @@ function ConsumerCard({ onStart, onStop, onDeploy, + outputWiring = [], + draggingOutput, + hoveredTarget, + onOutputWiringDelete, + envVarTargetRefs, + providers = [], }: ServiceCardProps) { const [isStarting, setIsStarting] = useState(false) const [showDeployMenu, setShowDeployMenu] = useState(false) @@ -1078,6 +1495,19 @@ function ConsumerCard({ })}
+ {/* Wirable Env Vars - drop targets for output wiring */} + {consumer.wirableEnvVars && consumer.wirableEnvVars.length > 0 && ( + + )} + {/* Service config vars at bottom */} {(missingRequiredVars.length > 0 || configuredVars.length > 0) && (
@@ -1230,3 +1660,204 @@ function CapabilitySlot({
) } + +// ============================================================================= +// Env Var Targets Section - Shows drop targets for output wiring +// ============================================================================= + +interface EnvVarTargetsSectionProps { + consumer: ConsumerInfo + outputWiring: OutputWiringInfo[] + draggingOutput?: OutputInfo | null + hoveredTarget?: string | null + onOutputWiringDelete?: (wiringId: string) => Promise + envVarTargetRefs?: Map + providers: ProviderInfo[] +} + +function EnvVarTargetsSection({ + consumer, + outputWiring, + draggingOutput, + hoveredTarget, + onOutputWiringDelete, + envVarTargetRefs, + providers, +}: EnvVarTargetsSectionProps) { + const envVars = consumer.wirableEnvVars || [] + if (envVars.length === 0) return null + + return ( +
+
+ + Wirable Env Vars +
+
+ {envVars.map((envVar) => { + // Find if this env var has a wired connection + const connection = outputWiring.find( + (w) => w.target_instance_id === consumer.id && w.target_env_var === envVar.key + ) + + // Get source info if connected + let sourceInfo: { instanceName: string; outputLabel: string } | null = null + if (connection) { + const sourceProvider = providers.find((p) => p.id === connection.source_instance_id) + const outputKey = connection.source_output_key + let outputLabel = outputKey + if (outputKey === 'access_url') { + outputLabel = 'URL' + } else if (outputKey.startsWith('env_vars.')) { + outputLabel = outputKey.replace('env_vars.', '') + } else if (outputKey.startsWith('capability_values.')) { + outputLabel = outputKey.replace('capability_values.', '') + } + sourceInfo = { + instanceName: sourceProvider?.name || connection.source_instance_id, + outputLabel, + } + } + + const targetId = `envvar::${consumer.id}::${envVar.key}` + const isDropTarget = draggingOutput !== null + const isHovered = hoveredTarget === targetId + + return ( + onOutputWiringDelete(connection.id) + : undefined + } + targetRef={(el) => { + if (el && envVarTargetRefs) { + envVarTargetRefs.set(targetId, el) + } + }} + targetId={targetId} + /> + ) + })} +
+
+ ) +} + +// ============================================================================= +// Env Var Drop Target Pill - Individual drop target for env vars +// ============================================================================= + +interface EnvVarDropTargetPillProps { + envVarKey: string + envVarLabel: string + value?: string + required?: boolean + isConnected: boolean + sourceInfo: { instanceName: string; outputLabel: string } | null + isDropTarget: boolean + isHovered: boolean + onDisconnect?: () => void + targetRef?: (el: HTMLDivElement | null) => void + targetId: string +} + +function EnvVarDropTargetPill({ + envVarKey, + envVarLabel, + value, + required, + isConnected, + sourceInfo, + isDropTarget, + isHovered, + onDisconnect, + targetRef, + targetId, +}: EnvVarDropTargetPillProps) { + return ( +
+ {/* Target indicator circle */} +
+ + {/* Content */} +
+
+ {required && *} + + {envVarLabel} + + ({envVarKey}) +
+ {isConnected && sourceInfo ? ( + + + + {sourceInfo.instanceName}.{sourceInfo.outputLabel} + + + ) : value ? ( + + Current: {value} + + ) : isDropTarget && !isConnected ? ( + + {isHovered ? 'Release to connect' : 'Drop output here'} + + ) : null} +
+ + {/* Disconnect button */} + {isConnected && onDisconnect && ( + + )} +
+ ) +} + diff --git a/ushadow/frontend/src/components/wiring/index.ts b/ushadow/frontend/src/components/wiring/index.ts index 799a8caf..daf3df3d 100644 --- a/ushadow/frontend/src/components/wiring/index.ts +++ b/ushadow/frontend/src/components/wiring/index.ts @@ -3,3 +3,14 @@ export { ServiceTemplateCard } from './ServiceTemplateCard' export { ServiceInstanceCard } from './ServiceInstanceCard' export { CapabilitySlot } from './CapabilitySlot' export { StatusIndicator } from './StatusIndicator' +export { + OutputPort, + EnvVarDropTarget, + WireOverlay, + OutputSection, + EnvVarSection, + type OutputInfo, + type EnvVarInfo, + type OutputWiringConnection, + type WirePosition, +} from './OutputWiring' diff --git a/ushadow/frontend/src/contexts/ChronicleContext.tsx b/ushadow/frontend/src/contexts/ChronicleContext.tsx index 7ea37021..ffebecdf 100644 --- a/ushadow/frontend/src/contexts/ChronicleContext.tsx +++ b/ushadow/frontend/src/contexts/ChronicleContext.tsx @@ -1,6 +1,6 @@ import { createContext, useContext, useState, useEffect, useCallback, ReactNode } from 'react' import { chronicleAuthApi } from '../services/chronicleApi' -import { useChronicleRecording, ChronicleRecordingReturn } from '../hooks/useChronicleRecording' +import { useWebRecording, WebRecordingReturn } from '../hooks/useWebRecording' interface ChronicleContextType { // Connection state @@ -13,7 +13,7 @@ interface ChronicleContextType { disconnect: () => void // Recording (lifted to context level for global access) - recording: ChronicleRecordingReturn + recording: WebRecordingReturn } const ChronicleContext = createContext(undefined) @@ -24,7 +24,7 @@ export function ChronicleProvider({ children }: { children: ReactNode }) { const [connectionError, setConnectionError] = useState(null) // Lift recording hook to context level - const recording = useChronicleRecording() + const recording = useWebRecording() // Check if Chronicle is connected (has valid auth token) const checkConnection = useCallback(async (): Promise => { diff --git a/ushadow/frontend/src/hooks/useChronicleRecording.ts b/ushadow/frontend/src/hooks/useWebRecording.ts similarity index 53% rename from ushadow/frontend/src/hooks/useChronicleRecording.ts rename to ushadow/frontend/src/hooks/useWebRecording.ts index cd171368..6652b917 100644 --- a/ushadow/frontend/src/hooks/useChronicleRecording.ts +++ b/ushadow/frontend/src/hooks/useWebRecording.ts @@ -1,14 +1,18 @@ /** - * Chronicle Recording Hook with Dual-Stream Support + * Web Recording Hook with Multi-Destination Support * * Supports three recording modes: * - 'streaming': Real-time microphone audio sent immediately * - 'batch': Microphone audio accumulated and sent when stopped * - 'dual-stream': Microphone + browser tab/screen audio mixed together + * + * Connects to wired audio consumers (Chronicle, Mycelia, etc.) based on + * the wiring configuration in the Service Configs page. */ import { useState, useRef, useCallback, useEffect } from 'react' import { getChronicleWebSocketUrl, getChronicleDirectUrl } from '../services/chronicleApi' +import { audioApi, AudioDestination } from '../services/api' import { getStorageKey } from '../utils/storage' import { useDualStreamRecording } from '../modules/dual-stream-audio/hooks/useDualStreamRecording' import { ChronicleWebSocketAdapter } from '../modules/dual-stream-audio/adapters/chronicleAdapter' @@ -26,7 +30,7 @@ export interface DebugStats { connectionAttempts: number } -export interface ChronicleRecordingReturn { +export interface WebRecordingReturn { // Current state currentStep: RecordingStep isRecording: boolean @@ -49,10 +53,19 @@ export interface ChronicleRecordingReturn { canAccessDualStream: boolean } -export const useChronicleRecording = (): ChronicleRecordingReturn => { +/** @deprecated Use useWebRecording instead */ +export type ChronicleRecordingReturn = WebRecordingReturn + +export const useWebRecording = (): WebRecordingReturn => { // Mode state const [mode, setMode] = useState('streaming') + // Keep mode ref in sync for use in callbacks + const currentModeRef = useRef(mode) + useEffect(() => { + currentModeRef.current = mode + }, [mode]) + // Debug stats const [debugStats, setDebugStats] = useState({ chunksSent: 0, @@ -66,6 +79,9 @@ export const useChronicleRecording = (): ChronicleRecordingReturn => { // Refs for WebSocket adapter and legacy mode const adapterRef = useRef(null) const legacyWsRef = useRef(null) + // Multi-destination WebSocket connections + const destinationWsRefs = useRef>(new Map()) + const activeDestinationsRef = useRef([]) const legacyStreamRef = useRef(null) const legacyContextRef = useRef(null) const legacyProcessorRef = useRef(null) @@ -74,6 +90,8 @@ export const useChronicleRecording = (): ChronicleRecordingReturn => { const keepAliveIntervalRef = useRef>() const chunkCountRef = useRef(0) const audioProcessingStartedRef = useRef(false) + // Batch mode: accumulate audio chunks to send all at once when stopping + const batchAudioChunksRef = useRef([]) // Legacy mode state (for streaming/batch modes) const [legacyStep, setLegacyStep] = useState('idle') @@ -159,6 +177,14 @@ export const useChronicleRecording = (): ChronicleRecordingReturn => { legacyWsRef.current = null } + // Close all destination WebSockets + destinationWsRefs.current.forEach((ws, id) => { + console.log(`Closing WebSocket for destination: ${id}`) + ws.close() + }) + destinationWsRefs.current.clear() + activeDestinationsRef.current = [] + if (durationIntervalRef.current) { clearInterval(durationIntervalRef.current) durationIntervalRef.current = undefined @@ -170,6 +196,7 @@ export const useChronicleRecording = (): ChronicleRecordingReturn => { } chunkCountRef.current = 0 + batchAudioChunksRef.current = [] }, []) // Start recording (dispatches based on mode) @@ -177,6 +204,7 @@ export const useChronicleRecording = (): ChronicleRecordingReturn => { try { // Reset state chunkCountRef.current = 0 + batchAudioChunksRef.current = [] setDebugStats(prev => ({ ...prev, chunksSent: 0, @@ -243,42 +271,167 @@ export const useChronicleRecording = (): ChronicleRecordingReturn => { }) legacyStreamRef.current = stream - // Connect WebSocket + // Fetch wired destinations from the audio provider API setLegacyStep('websocket') - const baseWsUrl = await getChronicleWebSocketUrl('/ws_pcm') - const wsUrl = `${baseWsUrl}?token=${token}&device_name=ushadow-recorder` + let destinations: AudioDestination[] = [] + let useRelay = false + let relayUrl: string | null = null + try { + const destResponse = await audioApi.getWiredDestinations() + destinations = destResponse.data.destinations || [] + useRelay = destResponse.data.use_relay || false + relayUrl = destResponse.data.relay_url || null + console.log('Wired audio destinations:', { destinations, useRelay, relayUrl }) + } catch (err) { + console.warn('Failed to fetch wired destinations, falling back to Chronicle:', err) + } + + // If no wired destinations, fall back to Chronicle (legacy behavior) + if (destinations.length === 0) { + try { + const baseWsUrl = await getChronicleWebSocketUrl('/ws_pcm') + destinations = [{ + consumer_id: 'chronicle', + consumer_name: 'Chronicle', + websocket_url: baseWsUrl, + protocol: 'wyoming', + format: 'pcm_s16le_16khz_mono' + }] + console.log('No wired destinations found, using Chronicle fallback') + } catch (err) { + throw new Error('No audio destinations wired. Please wire desktop-mic to a service like Chronicle or Mycelia in the Service Configs page.') + } + } - const ws = await new Promise((resolve, reject) => { - const socket = new WebSocket(wsUrl) + activeDestinationsRef.current = destinations + + // Connect to destinations - either via relay or directly + const connectedSockets: Map = new Map() + + if (useRelay && relayUrl) { + // RELAY MODE: Connect to single relay endpoint, backend handles fan-out + // Format destinations for relay: [{name: "Chronicle", url: "ws://..."}] + const relayDestinations = destinations.map(d => ({ + name: d.consumer_name, + url: d.websocket_url // Internal URLs from backend + })) + + const wsUrl = `${relayUrl}?destinations=${encodeURIComponent(JSON.stringify(relayDestinations))}&token=${token}` + console.log('Connecting via relay:', relayUrl) + console.log('Relay destinations:', relayDestinations) + + await new Promise((resolve, reject) => { + const socket = new WebSocket(wsUrl) + + socket.onopen = () => { + setTimeout(() => { + // In relay mode, use 'relay' as the socket ID + connectedSockets.set('relay', socket) + console.log('Connected to audio relay') + resolve() + }, 100) + } - socket.onopen = () => { - setTimeout(() => { - legacyWsRef.current = socket + socket.onerror = (e) => { + console.error('Relay connection error:', e) + reject(new Error('Failed to connect to audio relay')) + } - // Start keepalive - keepAliveIntervalRef.current = setInterval(() => { - if (socket.readyState === WebSocket.OPEN) { - socket.send(JSON.stringify({ type: 'ping', payload_length: null }) + '\n') + socket.onmessage = (event) => { + setDebugStats(prev => ({ ...prev, messagesReceived: prev.messagesReceived + 1 })) + // Log relay status messages + try { + const msg = JSON.parse(event.data) + if (msg.type === 'relay_status') { + console.log('Relay status:', msg.data) + } else if (msg.type === 'error') { + console.error('Relay error:', msg.message) } - }, 30000) - - resolve(socket) - }, 100) + } catch { + // Binary data, ignore + } + } + }) + + console.log('Connected via relay to destinations:', destinations.map(d => d.consumer_name).join(', ')) + + } else { + // DIRECT MODE: Connect to each destination individually + const connectionPromises = destinations.map(async (dest) => { + const wsUrl = `${dest.websocket_url}?token=${token}&device_name=ushadow-recorder` + console.log(`Connecting to ${dest.consumer_name}:`, wsUrl) + + return new Promise((resolve, reject) => { + const socket = new WebSocket(wsUrl) + + socket.onopen = () => { + setTimeout(() => { + connectedSockets.set(dest.consumer_id, socket) + console.log(`Connected to ${dest.consumer_name}`) + resolve() + }, 100) + } + + socket.onerror = () => { + console.warn(`Failed to connect to ${dest.consumer_name}`) + reject(new Error(`Failed to connect to ${dest.consumer_name}`)) + } + + socket.onmessage = () => { + setDebugStats(prev => ({ ...prev, messagesReceived: prev.messagesReceived + 1 })) + } + }) + }) + + // Wait for at least one connection to succeed + const results = await Promise.allSettled(connectionPromises) + const successfulConnections = results.filter(r => r.status === 'fulfilled').length + const failedConnections = results + .map((r, i) => r.status === 'rejected' ? destinations[i].consumer_name : null) + .filter(Boolean) + + if (successfulConnections === 0) { + const failedNames = failedConnections.join(', ') + throw new Error(`Failed to connect to audio destinations: ${failedNames}. Make sure the services are running.`) } - socket.onerror = () => reject(new Error('Failed to connect to Chronicle backend')) - socket.onmessage = () => { - setDebugStats(prev => ({ ...prev, messagesReceived: prev.messagesReceived + 1 })) + if (failedConnections.length > 0) { + console.warn(`Some audio destinations unavailable: ${failedConnections.join(', ')}`) } - }) + console.log(`Connected to ${successfulConnections}/${destinations.length} audio destinations`) + } - // Send audio-start + destinationWsRefs.current = connectedSockets + + // Use first connected socket for legacy compatibility + const firstSocket = connectedSockets.values().next().value + legacyWsRef.current = firstSocket + + // Start keepalive for all sockets + keepAliveIntervalRef.current = setInterval(() => { + destinationWsRefs.current.forEach((socket, id) => { + if (socket.readyState === WebSocket.OPEN) { + socket.send(JSON.stringify({ type: 'ping', payload_length: null }) + '\n') + } + }) + }, 30000) + + const ws = firstSocket! + + // Send audio-start to all connected destinations setLegacyStep('audio-start') - ws.send(JSON.stringify({ + const audioStartMsg = JSON.stringify({ type: 'audio-start', data: { rate: 16000, width: 2, channels: 1, mode }, payload_length: null - }) + '\n') + }) + '\n' + + destinationWsRefs.current.forEach((socket, id) => { + if (socket.readyState === WebSocket.OPEN) { + socket.send(audioStartMsg) + console.log(`Sent audio-start to ${id}`) + } + }) // Set up audio processing setLegacyStep('streaming') @@ -304,8 +457,8 @@ export const useChronicleRecording = (): ChronicleRecordingReturn => { processor.connect(audioContext.destination) processor.onaudioprocess = (event) => { - if (!ws || ws.readyState !== WebSocket.OPEN) return if (!audioProcessingStartedRef.current) return + if (destinationWsRefs.current.size === 0) return const inputData = event.inputBuffer.getChannelData(0) const pcmBuffer = new Int16Array(inputData.length) @@ -315,22 +468,42 @@ export const useChronicleRecording = (): ChronicleRecordingReturn => { pcmBuffer[i] = sample < 0 ? sample * 0x8000 : sample * 0x7FFF } - try { - if (ws.binaryType !== 'arraybuffer') { - ws.binaryType = 'arraybuffer' - } + // BATCH MODE: Accumulate chunks to send later + if (currentModeRef.current === 'batch') { + batchAudioChunksRef.current.push(pcmBuffer.slice()) // Clone the buffer + chunkCountRef.current++ + setDebugStats(prev => ({ ...prev, chunksSent: chunkCountRef.current })) + return + } - ws.send(JSON.stringify({ - type: 'audio-chunk', - data: { rate: 16000, width: 2, channels: 1 }, - payload_length: pcmBuffer.byteLength - }) + '\n') - ws.send(new Uint8Array(pcmBuffer.buffer, pcmBuffer.byteOffset, pcmBuffer.byteLength)) + // STREAMING MODE: Send immediately + const headerMsg = JSON.stringify({ + type: 'audio-chunk', + data: { rate: 16000, width: 2, channels: 1 }, + payload_length: pcmBuffer.byteLength + }) + '\n' + const audioData = new Uint8Array(pcmBuffer.buffer, pcmBuffer.byteOffset, pcmBuffer.byteLength) + + // Send to all connected destinations + let sentToAny = false + destinationWsRefs.current.forEach((socket) => { + try { + if (socket.readyState === WebSocket.OPEN) { + if (socket.binaryType !== 'arraybuffer') { + socket.binaryType = 'arraybuffer' + } + socket.send(headerMsg) + socket.send(audioData) + sentToAny = true + } + } catch (error) { + console.error('Failed to send audio chunk to destination:', error) + } + }) + if (sentToAny) { chunkCountRef.current++ setDebugStats(prev => ({ ...prev, chunksSent: chunkCountRef.current })) - } catch (error) { - console.error('Failed to send audio chunk:', error) } } @@ -392,14 +565,53 @@ export const useChronicleRecording = (): ChronicleRecordingReturn => { // Stop legacy recording audioProcessingStartedRef.current = false - if (legacyWsRef.current?.readyState === WebSocket.OPEN) { - legacyWsRef.current.send(JSON.stringify({ - type: 'audio-stop', - data: { timestamp: Date.now() }, - payload_length: null - }) + '\n') + // BATCH MODE: Send all accumulated audio chunks before stopping + if (currentModeRef.current === 'batch' && batchAudioChunksRef.current.length > 0) { + console.log(`Sending ${batchAudioChunksRef.current.length} accumulated batch chunks`) + + // Send each accumulated chunk to all destinations + for (const pcmBuffer of batchAudioChunksRef.current) { + const headerMsg = JSON.stringify({ + type: 'audio-chunk', + data: { rate: 16000, width: 2, channels: 1 }, + payload_length: pcmBuffer.byteLength + }) + '\n' + const audioData = new Uint8Array(pcmBuffer.buffer, pcmBuffer.byteOffset, pcmBuffer.byteLength) + + destinationWsRefs.current.forEach((socket) => { + try { + if (socket.readyState === WebSocket.OPEN) { + if (socket.binaryType !== 'arraybuffer') { + socket.binaryType = 'arraybuffer' + } + socket.send(headerMsg) + socket.send(audioData) + } + } catch (error) { + console.error('Failed to send batch audio chunk:', error) + } + }) + } + + console.log('Finished sending batch audio') + // Clear the batch buffer + batchAudioChunksRef.current = [] } + // Send audio-stop to all connected destinations + const audioStopMsg = JSON.stringify({ + type: 'audio-stop', + data: { timestamp: Date.now() }, + payload_length: null + }) + '\n' + + destinationWsRefs.current.forEach((socket, id) => { + if (socket.readyState === WebSocket.OPEN) { + socket.send(audioStopMsg) + console.log(`Sent audio-stop to ${id}`) + } + }) + legacyCleanup() setLegacyRecording(false) @@ -449,3 +661,6 @@ export const useChronicleRecording = (): ChronicleRecordingReturn => { canAccessDualStream } } + +/** @deprecated Use useWebRecording instead */ +export const useChronicleRecording = useWebRecording diff --git a/ushadow/frontend/src/modules/dual-stream-audio/adapters/multiDestinationAdapter.ts b/ushadow/frontend/src/modules/dual-stream-audio/adapters/multiDestinationAdapter.ts new file mode 100644 index 00000000..f1524cee --- /dev/null +++ b/ushadow/frontend/src/modules/dual-stream-audio/adapters/multiDestinationAdapter.ts @@ -0,0 +1,237 @@ +/** + * Multi-Destination WebSocket Adapter + * + * Sends audio to the relay endpoint which fans out to multiple consumers + * (Chronicle, Mycelia, etc.) simultaneously. + * + * Uses the /ws/audio/relay endpoint on ushadow backend. + */ + +import type { AudioChunk, RecordingMode } from '../core/types' + +export interface AudioDestination { + name: string + url: string +} + +export interface MultiDestinationConfig { + relayUrl: string // e.g., "ws://localhost:8000/ws/audio/relay" + token: string + destinations: AudioDestination[] + deviceName?: string + mode?: RecordingMode +} + +export interface DestinationStatus { + name: string + connected: boolean + errors: number +} + +export class MultiDestinationAdapter { + private ws: WebSocket | null = null + private config: MultiDestinationConfig + private isConnected: boolean = false + private messageQueue: any[] = [] + private destinationStatus: DestinationStatus[] = [] + + // Callbacks + onStatusChange?: (status: DestinationStatus[]) => void + onError?: (error: Error) => void + + constructor(config: MultiDestinationConfig) { + this.config = config + } + + /** + * Connect to the relay WebSocket + */ + async connect(): Promise { + return new Promise((resolve, reject) => { + const { relayUrl, token, destinations, deviceName = 'webui-dual-stream' } = this.config + + // Build relay URL with destinations parameter + const destinationsParam = encodeURIComponent(JSON.stringify(destinations)) + const wsUrl = `${relayUrl}?destinations=${destinationsParam}&token=${token}&device_name=${deviceName}` + + console.log('🔗 Connecting to Multi-Destination Relay:', wsUrl) + console.log('📍 Destinations:', destinations.map(d => d.name).join(', ')) + + this.ws = new WebSocket(wsUrl) + + this.ws.onopen = () => { + console.log('✅ Multi-Destination Relay connected') + + // Send stabilization delay + setTimeout(() => { + this.isConnected = true + + // Flush any queued messages + while (this.messageQueue.length > 0) { + const msg = this.messageQueue.shift() + this.ws?.send(msg) + } + + resolve() + }, 100) + } + + this.ws.onmessage = (event) => { + try { + const data = JSON.parse(event.data) + + // Handle relay status updates + if (data.type === 'relay_status') { + this.handleRelayStatus(data.data) + } else if (data.type === 'error') { + console.error('❌ Relay error:', data.message) + this.onError?.(new Error(data.message)) + } + } catch (e) { + // Non-JSON message, ignore + } + } + + this.ws.onerror = (error) => { + console.error('❌ Multi-Destination Relay error:', error) + reject(new Error('WebSocket connection failed')) + } + + this.ws.onclose = (event) => { + console.log('🔌 Multi-Destination Relay closed:', event.code, event.reason) + this.isConnected = false + } + }) + } + + /** + * Handle relay status updates + */ + private handleRelayStatus(data: any) { + if (data.destinations) { + this.destinationStatus = data.destinations.map((d: any) => ({ + name: d.name, + connected: d.connected, + errors: d.errors || 0, + })) + + console.log('📊 Relay status:', this.destinationStatus) + this.onStatusChange?.(this.destinationStatus) + } + } + + /** + * Send audio-start message (Wyoming protocol) + */ + sendAudioStart(sampleRate: number = 16000, channels: number = 1): void { + const message = JSON.stringify({ + type: 'audio-start', + data: { + rate: sampleRate, + width: 2, // 16-bit + channels: channels, + mode: this.config.mode || 'streaming', + } + }) + '\n' + + if (this.isConnected && this.ws) { + this.ws.send(message) + } else { + this.messageQueue.push(message) + } + } + + /** + * Send audio chunk (binary PCM data) + */ + sendAudioChunk(chunk: AudioChunk): void { + if (!this.isConnected || !this.ws) { + console.warn('⚠️ Cannot send audio: not connected') + return + } + + // Send raw PCM bytes + this.ws.send(chunk.data) + } + + /** + * Send audio-stop message + */ + sendAudioStop(): void { + const message = JSON.stringify({ + type: 'audio-stop', + data: { + timestamp: Date.now(), + } + }) + '\n' + + if (this.isConnected && this.ws) { + this.ws.send(message) + } + } + + /** + * Disconnect from relay + */ + disconnect(): void { + if (this.ws) { + this.sendAudioStop() + this.ws.close() + this.ws = null + } + this.isConnected = false + } + + /** + * Get current destination status + */ + getDestinationStatus(): DestinationStatus[] { + return this.destinationStatus + } + + /** + * Check if connected + */ + get connected(): boolean { + return this.isConnected + } +} + +/** + * Create adapter from audio consumer config + * + * Usage: + * ```typescript + * // Fetch config from API + * const consumer = await getActiveAudioConsumer(baseUrl, token); + * + * // Create adapter + * const adapter = createMultiDestinationAdapter(consumer, token); + * await adapter.connect(); + * + * // Start recording + * adapter.sendAudioStart(); + * // ... send chunks ... + * adapter.sendAudioStop(); + * adapter.disconnect(); + * ``` + */ +export function createMultiDestinationAdapter( + consumerConfig: { + websocket_url: string + destinations?: AudioDestination[] + }, + token: string, + options?: { + deviceName?: string + mode?: RecordingMode + } +): MultiDestinationAdapter { + return new MultiDestinationAdapter({ + relayUrl: consumerConfig.websocket_url, + token, + destinations: consumerConfig.destinations || [], + deviceName: options?.deviceName, + mode: options?.mode, + }) +} diff --git a/ushadow/frontend/src/pages/ServiceConfigsPage.tsx b/ushadow/frontend/src/pages/ServiceConfigsPage.tsx index 0e1bde90..a41bb7a4 100644 --- a/ushadow/frontend/src/pages/ServiceConfigsPage.tsx +++ b/ushadow/frontend/src/pages/ServiceConfigsPage.tsx @@ -181,6 +181,10 @@ export default function ServiceConfigsPage() { console.log('Templates loaded:', templatesRes.data) console.log('Compose templates (before filter):', templatesRes.data.filter((t: any) => t.source === 'compose')) console.log('Compose templates (after installed filter):', templatesRes.data.filter((t: any) => t.source === 'compose' && t.installed)) + // Debug: show requires for each compose template + templatesRes.data.filter((t: any) => t.source === 'compose').forEach((t: any) => { + console.log(` ${t.id}: installed=${t.installed}, requires=${JSON.stringify(t.requires)}`) + }) setTemplates(templatesRes.data) setServiceConfigs(instancesRes.data) @@ -550,8 +554,11 @@ export default function ServiceConfigsPage() { // Consumer/Service handlers for WiringBoard const handleStartConsumer = async (consumerId: string) => { try { + // Find the consumer to get its templateId (instances have different id vs templateId) + const consumer = wiringConsumers.find(c => c.id === consumerId) + const templateId = consumer?.templateId || consumerId // Extract service name from template ID (format: "compose_file:service_name") - const serviceName = consumerId.includes(':') ? consumerId.split(':').pop()! : consumerId + const serviceName = templateId.includes(':') ? templateId.split(':').pop()! : templateId await servicesApi.startService(serviceName) setMessage({ type: 'success', text: `${consumerId} started` }) // Reload service statuses @@ -567,8 +574,11 @@ export default function ServiceConfigsPage() { const handleStopConsumer = async (consumerId: string) => { try { + // Find the consumer to get its templateId (instances have different id vs templateId) + const consumer = wiringConsumers.find(c => c.id === consumerId) + const templateId = consumer?.templateId || consumerId // Extract service name from template ID (format: "compose_file:service_name") - const serviceName = consumerId.includes(':') ? consumerId.split(':').pop()! : consumerId + const serviceName = templateId.includes(':') ? templateId.split(':').pop()! : templateId await servicesApi.stopService(serviceName) setMessage({ type: 'success', text: `${consumerId} stopped` }) // Reload service statuses @@ -871,9 +881,9 @@ export default function ServiceConfigsPage() { .filter((t) => t.source === 'provider' && t.provides) const wiringProviders = [ - // Templates (defaults) - only show configured ones + // Templates (defaults) - show configured ones OR client/upload/remote mode (no config needed) ...providerTemplates - .filter((t) => t.configured) // Only show providers that have been set up + .filter((t) => t.configured || ['client', 'upload', 'remote', 'relay'].includes(t.mode)) // Client-mode providers don't need setup .map((t) => { // Extract config vars from schema - include all fields with required indicator const configVars: Array<{ key: string; label: string; value: string; isSecret: boolean; required?: boolean }> = diff --git a/ushadow/frontend/src/pages/ServicesPage.tsx b/ushadow/frontend/src/pages/ServicesPage.tsx index 611b0848..f8633bce 100644 --- a/ushadow/frontend/src/pages/ServicesPage.tsx +++ b/ushadow/frontend/src/pages/ServicesPage.tsx @@ -1,4 +1,5 @@ import { useState, useEffect } from 'react' +import { useNavigate } from 'react-router-dom' import { createPortal } from 'react-dom' import { Server, @@ -39,17 +40,19 @@ import ImportFromGitHubModal from '../components/ImportFromGitHubModal' import { StatusBadge } from '../components/StatusBadge' export default function ServicesPage() { + const navigate = useNavigate() + // Compose services state const [services, setServices] = useState([]) const [serviceStatuses, setServiceStatuses] = useState>({}) const [expandedServices, setExpandedServices] = useState>(new Set()) const [editingServiceId, setEditingServiceId] = useState(null) - const [envConfig, setEnvConfig] = useState<{ + const [envConfigs, setEnvConfigs] = useState(null) - const [envEditForm, setEnvEditForm] = useState>({}) - const [customEnvVars, setCustomEnvVars] = useState>([]) + }>>({}) + const [envEditForms, setEnvEditForms] = useState>>({}) + const [customEnvVars, setCustomEnvVars] = useState>>({}) // Provider state const [capabilities, setCapabilities] = useState([]) @@ -390,11 +393,13 @@ export default function ServicesPage() { const handleSaveEnvVars = async (serviceId: string) => { setSaving(true) try { + const serviceEnvEditForm = envEditForms[serviceId] || {} + const serviceCustomEnvVars = customEnvVars[serviceId] || [] // Combine standard env vars with custom ones const envVars: EnvVarConfig[] = [ - ...Object.values(envEditForm), + ...Object.values(serviceEnvEditForm), // Add custom env vars as new settings - ...customEnvVars + ...serviceCustomEnvVars .filter(ev => ev.name.trim() && ev.value.trim()) .map(ev => ({ name: ev.name.trim().toUpperCase(), @@ -407,7 +412,7 @@ export default function ServicesPage() { const result = await servicesApi.updateEnvConfig(serviceId, envVars) console.log('Save result:', result) const newSettingsCount = (result.data as any)?.new_settings_created || 0 - const customCount = customEnvVars.filter(ev => ev.name.trim() && ev.value.trim()).length + const customCount = serviceCustomEnvVars.filter(ev => ev.name.trim() && ev.value.trim()).length let msg = 'Environment configuration saved' if (newSettingsCount > 0 || customCount > 0) { const total = newSettingsCount + customCount @@ -415,9 +420,22 @@ export default function ServicesPage() { } setMessage({ type: 'success', text: msg }) setEditingServiceId(null) - setEnvConfig(null) - setEnvEditForm({}) - setCustomEnvVars([]) + // Clear this service's config + setEnvConfigs(prev => { + const next = { ...prev } + delete next[serviceId] + return next + }) + setEnvEditForms(prev => { + const next = { ...prev } + delete next[serviceId] + return next + }) + setCustomEnvVars(prev => { + const next = { ...prev } + delete next[serviceId] + return next + }) // Reload services to update needs_setup status const servicesRes = await servicesApi.getInstalled() setServices(servicesRes.data) @@ -429,10 +447,25 @@ export default function ServicesPage() { } const handleCancelEnvEdit = () => { + if (editingServiceId) { + // Clear only the editing service's config + setEnvConfigs(prev => { + const next = { ...prev } + delete next[editingServiceId] + return next + }) + setEnvEditForms(prev => { + const next = { ...prev } + delete next[editingServiceId] + return next + }) + setCustomEnvVars(prev => { + const next = { ...prev } + delete next[editingServiceId] + return next + }) + } setEditingServiceId(null) - setEnvConfig(null) - setEnvEditForm({}) - setCustomEnvVars([]) } const handleExpandService = async (serviceId: string) => { @@ -456,8 +489,8 @@ export default function ServicesPage() { } }) - setEnvConfig(data) - setEnvEditForm(formData) + setEnvConfigs(prev => ({ ...prev, [serviceId]: data })) + setEnvEditForms(prev => ({ ...prev, [serviceId]: formData })) } catch (error: any) { setMessage({ type: 'error', text: 'Failed to load env configuration' }) } finally { @@ -474,15 +507,31 @@ export default function ServicesPage() { // Clear edit state if collapsing if (editingServiceId === serviceId) { setEditingServiceId(null) - setEnvConfig(null) - setEnvEditForm({}) + setEnvConfigs(prev => { + const next = { ...prev } + delete next[serviceId] + return next + }) + setEnvEditForms(prev => { + const next = { ...prev } + delete next[serviceId] + return next + }) + setCustomEnvVars(prev => { + const next = { ...prev } + delete next[serviceId] + return next + }) } } - const updateEnvVar = (name: string, updates: Partial) => { - setEnvEditForm(prev => ({ + const updateEnvVar = (serviceId: string, name: string, updates: Partial) => { + setEnvEditForms(prev => ({ ...prev, - [name]: { ...prev[name], ...updates } + [serviceId]: { + ...(prev[serviceId] || {}), + [name]: { ...(prev[serviceId]?.[name] || {}), ...updates } + } })) } @@ -910,6 +959,9 @@ export default function ServicesPage() { const isEditing = editingServiceId === service.service_id const isStarting = startingService === service.service_name const isLoadingConfig = loadingEnvConfig === service.service_id + const envConfig = envConfigs[service.service_id] + const envEditForm = envEditForms[service.service_id] || {} + const serviceCustomEnvVars = customEnvVars[service.service_id] || [] // Debug logging if (isExpanded || isLoadingConfig) { @@ -973,10 +1025,16 @@ export default function ServicesPage() { +
+ ) : ( +
+
+ +
+

+ Token Saved +

+

+ Your Mycelia authentication credentials have been saved securely. +

+
+
+
+ )} +
+ ) +} + +interface CompleteStepProps { + tokenData: { token: string; clientId: string } | null +} + +function CompleteStep({ tokenData }: CompleteStepProps) { + return ( +
+
+
+ +
+
+

+ Mycelia is Ready! +

+

+ Your AI memory and timeline service is configured and ready to start. +

+
+
+ +
+

+ What's Next? +

+
    +
  • + + Access the web UI at https://localhost:14433 +
    • +
  • + + Connect Apple Voice Memos, Google Drive, or local audio files +
    • +
  • + + Search your voice notes and conversations +
    • +
+
+ + {tokenData && ( +
+
+ +
+

Credentials saved

+

Your token and client ID have been saved to ushadow settings and will be automatically passed to Mycelia.

+
+
+
+ )} +
+ ) +} diff --git a/ushadow/frontend/src/wizards/index.ts b/ushadow/frontend/src/wizards/index.ts index 46a0db45..7804c0a6 100644 --- a/ushadow/frontend/src/wizards/index.ts +++ b/ushadow/frontend/src/wizards/index.ts @@ -14,6 +14,7 @@ export { default as QuickstartWizard } from './QuickstartWizard' export { default as LocalServicesWizard } from './LocalServicesWizard' export { default as MobileAppWizard } from './MobileAppWizard' export { default as SpeakerRecognitionWizard } from './SpeakerRecognitionWizard' +export { default as MyceliaWizard } from './MyceliaWizard' // Export wizard registry for dynamic discovery export { wizardRegistry, getAllWizards, getWizardById } from './registry' diff --git a/ushadow/frontend/src/wizards/registry.ts b/ushadow/frontend/src/wizards/registry.ts index d8de0a82..75ed584c 100644 --- a/ushadow/frontend/src/wizards/registry.ts +++ b/ushadow/frontend/src/wizards/registry.ts @@ -5,7 +5,7 @@ * Used by WizardStartPage to dynamically list all wizards. */ -import { LucideIcon, Sparkles, Shield, Smartphone, Mic, CheckCircle2, Wand2, Server, MessageSquare, Brain } from 'lucide-react' +import { LucideIcon, Sparkles, Shield, Smartphone, Mic, CheckCircle2, Wand2, Server, MessageSquare, Brain, Database } from 'lucide-react' export interface WizardMetadata { id: string @@ -88,6 +88,13 @@ export const wizardRegistry: WizardMetadata[] = [ description: 'OpenMemory setup', icon: Brain, }, + { + id: 'mycelia', + path: '/wizard/mycelia', + label: 'Mycelia', + description: 'AI memory and timeline', + icon: Database, + }, ] /** Icon to show when all wizards are complete */ diff --git a/ushadow/mobile/MULTI_DESTINATION_AUDIO_EXAMPLE.md b/ushadow/mobile/MULTI_DESTINATION_AUDIO_EXAMPLE.md new file mode 100644 index 00000000..937e35aa --- /dev/null +++ b/ushadow/mobile/MULTI_DESTINATION_AUDIO_EXAMPLE.md @@ -0,0 +1,186 @@ +# Multi-Destination Audio Streaming + +Stream audio from mobile app to multiple destinations (Chronicle + Mycelia) simultaneously. + +## Architecture Options + +### Option 1: Client-Side Multi-Cast (Direct) +``` +Mobile App + ├─> WebSocket 1 → Chronicle /chronicle/ws_pcm + └─> WebSocket 2 → Mycelia /ws_pcm +``` + +**Pros**: Simple, no server changes +**Cons**: 2x bandwidth, 2x battery usage + +### Option 2: Server-Side Relay (Recommended) ✨ +``` +Mobile App → WebSocket → Relay Server + ├─> Chronicle /chronicle/ws_pcm + └─> Mycelia /ws_pcm +``` + +**Pros**: +- Single connection from mobile (saves battery/bandwidth) +- Centralized error handling +- Can add destinations without mobile app changes + +**Cons**: Requires server relay endpoint + +## Usage Example + +```typescript +import { useRelayStreamer } from './hooks/useMultiDestinationStreamer'; +import { usePhoneAudioRecorder } from './hooks/usePhoneAudioRecorder'; + +function MultiDestinationRecording() { + const relay = useRelayStreamer(); + const recorder = usePhoneAudioRecorder(); + + const startRecording = async () => { + // Configure destinations + const config = { + mode: 'relay' as const, + relayUrl: 'wss://your-ushadow-host.ts.net/ws/audio/relay', + destinations: [ + { + name: 'chronicle', + url: 'ws://localhost:5001/chronicle/ws_pcm' + }, + { + name: 'mycelia', + url: 'ws://localhost:5173/ws_pcm' + } + ] + }; + + // Start relay connection + await relay.startMultiStreaming(config, 'streaming'); + + // Start recording and connect audio pipeline + await recorder.startRecording((audioData) => { + // Send to relay (which forwards to both destinations) + relay.sendAudioToAll(new Uint8Array(audioData)); + }); + }; + + const stopRecording = async () => { + await recorder.stopRecording(); + relay.stopMultiStreaming(); + }; + + return ( + + + + + + {/* Status for each destination */} + {Object.entries(relay.destinationStatus).map(([name, status]) => ( + + {name}: {status.isStreaming ? '✓' : '✗'} + {status.error && {status.error}} + + ))} + + ); +} +``` + +## Configuration via UNode Settings + +You can store multi-destination config in UNode settings: + +```typescript +// In unode-details.tsx or similar +const [multiDestEnabled, setMultiDestEnabled] = useState(false); +const [destinations, setDestinations] = useState([ + { name: 'chronicle', url: 'ws://localhost:5001/chronicle/ws_pcm', enabled: true }, + { name: 'mycelia', url: 'ws://localhost:5173/ws_pcm', enabled: false }, +]); + +// Save to UNode storage +await AsyncStorage.setItem( + `unode_${unodeId}_multi_dest`, + JSON.stringify({ enabled: multiDestEnabled, destinations }) +); +``` + +## Testing the Relay + +1. **Start ushadow backend** with relay endpoint: + ```bash + cd ushadow/backend + uvicorn main:app --host 0.0.0.0 --port 8000 + ``` + +2. **Start Chronicle** (if using): + ```bash + cd compose + docker-compose -f chronicle-compose.yaml up + ``` + +3. **Start Mycelia**: + ```bash + cd mycelia/backend + deno run -A server.ts serve --port 5173 + ``` + +4. **Test relay endpoint**: + ```bash + # Check relay status + curl http://localhost:8000/ws/audio/relay/status + + # Response: + { + "endpoint": "/ws/audio/relay", + "protocol": "Wyoming", + "description": "Multi-destination audio relay" + } + ``` + +5. **Connect from mobile**: + ```typescript + const relayUrl = 'ws://localhost:8000/ws/audio/relay'; + // or via Tailscale: + const relayUrl = 'wss://your-machine.ts.net/ws/audio/relay'; + ``` + +## Security Notes + +- Relay endpoint requires JWT token authentication +- Token is passed to destination endpoints for their auth +- All WebSocket connections use secure WebSocket (wss://) in production +- Consider rate limiting on relay endpoint + +## Performance + +**Bandwidth comparison** (16kHz, 16-bit, mono): +- Direct multi-cast: 32 KB/s × 2 = **64 KB/s** +- Server relay: **32 KB/s** (mobile → server) + - Server → destinations: 32 KB/s × 2 (server bandwidth) + +**Battery impact**: +- Direct: 2x WebSocket connections = higher battery drain +- Relay: 1x WebSocket connection = standard battery usage + +## Future Enhancements + +1. **Dynamic destination management**: Add/remove destinations during streaming +2. **Per-destination settings**: Different audio formats or modes per destination +3. **Fallback handling**: Continue streaming if one destination fails +4. **Metrics/monitoring**: Track relay performance and destination health +5. **Compression**: Compress audio before relay to reduce bandwidth diff --git a/ushadow/mobile/app/hooks/useMultiDestinationStreamer.ts b/ushadow/mobile/app/hooks/useMultiDestinationStreamer.ts new file mode 100644 index 00000000..144e103e --- /dev/null +++ b/ushadow/mobile/app/hooks/useMultiDestinationStreamer.ts @@ -0,0 +1,186 @@ +/** + * useMultiDestinationStreamer.ts + * + * Hook for streaming audio to multiple destinations via relay server. + * + * Two modes: + * 1. Direct multi-cast: Opens multiple WebSocket connections from mobile + * 2. Server relay: Single connection to relay server that fans out + */ +import { useState, useCallback } from 'react'; +import { useAudioStreamer } from './useAudioStreamer'; + +export interface MultiDestinationConfig { + destinations: Array<{ + name: string; + url: string; + }>; + mode: 'direct' | 'relay'; + relayUrl?: string; // Required if mode === 'relay' +} + +export interface UseMultiDestinationStreamer { + isStreaming: boolean; + startMultiStreaming: ( + config: MultiDestinationConfig, + streamMode?: 'batch' | 'streaming' + ) => Promise; + stopMultiStreaming: () => void; + sendAudioToAll: (audioBytes: Uint8Array) => void; + destinationStatus: Record; +} + +/** + * Direct multi-cast: Opens multiple WebSocket connections + */ +export const useDirectMultiCast = (): UseMultiDestinationStreamer => { + const [streamers, setStreamers] = useState>>({}); + const [isStreaming, setIsStreaming] = useState(false); + + const startMultiStreaming = useCallback(async ( + config: MultiDestinationConfig, + streamMode: 'batch' | 'streaming' = 'streaming' + ) => { + // Create streamer for each destination + const newStreamers: Record = {}; + for (const dest of config.destinations) { + // This is a simplified version - in real implementation, + // you'd need to dynamically create useAudioStreamer instances + // which is tricky with React hooks. See relay mode for better approach. + console.warn('[DirectMultiCast] Direct mode requires pre-configured streamers'); + } + setIsStreaming(true); + }, []); + + const stopMultiStreaming = useCallback(() => { + Object.values(streamers).forEach(streamer => streamer.stopStreaming()); + setIsStreaming(false); + }, [streamers]); + + const sendAudioToAll = useCallback((audioBytes: Uint8Array) => { + Object.values(streamers).forEach(streamer => streamer.sendAudio(audioBytes)); + }, [streamers]); + + const destinationStatus = Object.entries(streamers).reduce((acc, [name, streamer]) => { + acc[name] = { + isStreaming: streamer.isStreaming, + error: streamer.error, + }; + return acc; + }, {} as Record); + + return { + isStreaming, + startMultiStreaming, + stopMultiStreaming, + sendAudioToAll, + destinationStatus, + }; +}; + +/** + * Server relay: Single connection to relay endpoint + * RECOMMENDED - more efficient for mobile + */ +export const useRelayStreamer = (): UseMultiDestinationStreamer => { + const relayStreamer = useAudioStreamer(); + const [destinationStatus, setDestinationStatus] = useState< + Record + >({}); + + const startMultiStreaming = useCallback(async ( + config: MultiDestinationConfig, + streamMode: 'batch' | 'streaming' = 'streaming' + ) => { + if (config.mode !== 'relay' || !config.relayUrl) { + throw new Error('Relay mode requires relayUrl'); + } + + // Build relay URL with destinations parameter + const destinationsParam = encodeURIComponent(JSON.stringify(config.destinations)); + + // Get token from storage or context + // TODO: Replace with actual token retrieval + const token = 'YOUR_JWT_TOKEN'; + + const relayWsUrl = `${config.relayUrl}?destinations=${destinationsParam}&token=${token}`; + + console.log('[RelayStreamer] Connecting to relay:', relayWsUrl); + + // Connect to relay server + await relayStreamer.startStreaming(relayWsUrl, streamMode); + + // Initialize status for all destinations + const status: Record = {}; + config.destinations.forEach(dest => { + status[dest.name] = { isStreaming: true, error: null }; + }); + setDestinationStatus(status); + }, [relayStreamer]); + + const stopMultiStreaming = useCallback(() => { + relayStreamer.stopStreaming(); + setDestinationStatus({}); + }, [relayStreamer]); + + const sendAudioToAll = useCallback((audioBytes: Uint8Array) => { + // Send once to relay, it handles fanout + relayStreamer.sendAudio(audioBytes); + }, [relayStreamer]); + + return { + isStreaming: relayStreamer.isStreaming, + startMultiStreaming, + stopMultiStreaming, + sendAudioToAll, + destinationStatus, + }; +}; + +/** + * Main hook - auto-selects mode based on config + */ +export const useMultiDestinationStreamer = (): UseMultiDestinationStreamer => { + const relayStreamer = useRelayStreamer(); + const directStreamer = useDirectMultiCast(); + const [currentMode, setCurrentMode] = useState<'direct' | 'relay'>('relay'); + + const startMultiStreaming = useCallback(async ( + config: MultiDestinationConfig, + streamMode: 'batch' | 'streaming' = 'streaming' + ) => { + setCurrentMode(config.mode); + + if (config.mode === 'relay') { + return relayStreamer.startMultiStreaming(config, streamMode); + } else { + return directStreamer.startMultiStreaming(config, streamMode); + } + }, [relayStreamer, directStreamer]); + + const stopMultiStreaming = useCallback(() => { + if (currentMode === 'relay') { + relayStreamer.stopMultiStreaming(); + } else { + directStreamer.stopMultiStreaming(); + } + }, [currentMode, relayStreamer, directStreamer]); + + const sendAudioToAll = useCallback((audioBytes: Uint8Array) => { + if (currentMode === 'relay') { + relayStreamer.sendAudioToAll(audioBytes); + } else { + directStreamer.sendAudioToAll(audioBytes); + } + }, [currentMode, relayStreamer, directStreamer]); + + const activeStreamer = currentMode === 'relay' ? relayStreamer : directStreamer; + + return { + isStreaming: activeStreamer.isStreaming, + startMultiStreaming, + stopMultiStreaming, + sendAudioToAll, + destinationStatus: activeStreamer.destinationStatus, + }; +}; diff --git a/ushadow/mobile/app/services/audioProviderApi.ts b/ushadow/mobile/app/services/audioProviderApi.ts new file mode 100644 index 00000000..03b7a0ff --- /dev/null +++ b/ushadow/mobile/app/services/audioProviderApi.ts @@ -0,0 +1,141 @@ +/** + * audioProviderApi.ts + * + * API client for fetching audio consumer configuration. + * Mobile app is the audio INPUT (provider/source). + * This API tells the mobile app WHERE to send audio (the consumer/destination). + */ + +export interface AudioConsumerConfig { + provider_id: string; + name: string; + websocket_url: string; + protocol: string; + format: string; + mode?: string; + destinations?: Array<{ name: string; url: string }>; +} + +export interface AudioConsumerResponse { + capability: 'audio_consumer'; + selected_provider: string; + config: AudioConsumerConfig; + available_providers: string[]; +} + +/** + * Fetch the active audio consumer configuration. + * This tells the mobile app WHERE to send its audio. + */ +export async function getActiveAudioConsumer( + baseUrl: string, + token: string +): Promise { + const url = `${baseUrl}/api/providers/audio_consumer/active`; + + const response = await fetch(url, { + method: 'GET', + headers: { + 'Authorization': `Bearer ${token}`, + 'Content-Type': 'application/json', + }, + }); + + if (!response.ok) { + throw new Error(`Failed to fetch audio consumer: ${response.statusText}`); + } + + const data: AudioConsumerResponse = await response.json(); + return data.config; +} + +/** + * Get available audio consumers (Chronicle, Mycelia, etc.) + */ +export async function getAvailableAudioConsumers( + baseUrl: string, + token: string +): Promise> { + const url = `${baseUrl}/api/providers/audio_consumer/available`; + + const response = await fetch(url, { + method: 'GET', + headers: { + 'Authorization': `Bearer ${token}`, + 'Content-Type': 'application/json', + }, + }); + + if (!response.ok) { + throw new Error(`Failed to fetch available consumers: ${response.statusText}`); + } + + const data = await response.json(); + return data.providers || []; +} + +/** + * Set the active audio consumer (requires admin permission) + * Changes where mobile audio gets sent (Chronicle, Mycelia, etc.) + */ +export async function setActiveAudioConsumer( + baseUrl: string, + token: string, + consumerId: string +): Promise { + const url = `${baseUrl}/api/providers/audio_consumer/active`; + + const response = await fetch(url, { + method: 'PUT', + headers: { + 'Authorization': `Bearer ${token}`, + 'Content-Type': 'application/json', + }, + body: JSON.stringify({ + provider_id: consumerId, + }), + }); + + if (!response.ok) { + throw new Error(`Failed to set audio consumer: ${response.statusText}`); + } +} + +/** + * Build full WebSocket URL with token + */ +export function buildAudioStreamUrl( + config: AudioConsumerConfig, + token: string +): string { + const url = new URL(config.websocket_url); + + // Add token as query parameter + url.searchParams.set('token', token); + + // For multi-destination relay, add destinations + if (config.provider_id === 'multi-destination' && config.destinations) { + url.searchParams.set( + 'destinations', + JSON.stringify(config.destinations) + ); + } + + return url.toString(); +} + +/** + * Example usage in mobile app: + * + * // Mobile app is the audio INPUT source + * // This API tells it WHERE to send audio (the consumer) + * + * const consumer = await getActiveAudioConsumer('https://ushadow.ts.net', jwtToken); + * // Returns: { provider_id: "chronicle", websocket_url: "ws://chronicle:5001/chronicle/ws_pcm", ... } + * + * const wsUrl = buildAudioStreamUrl(consumer, jwtToken); + * // Result: "ws://chronicle:5001/chronicle/ws_pcm?token=JWT" + * + * await audioStreamer.startStreaming(wsUrl, 'streaming'); + * // Mobile mic → Chronicle + */