diff --git a/.env.example b/.env.example new file mode 100644 index 00000000..4136a0a5 --- /dev/null +++ b/.env.example @@ -0,0 +1,55 @@ +# Ushadow Environment Configuration Template +# Copy this file to .env and customize for your environment +# DO NOT COMMIT .env - it contains environment-specific configuration + +# ========================================== +# ENVIRONMENT & PROJECT NAMING +# ========================================== +ENV_NAME=ushadow +COMPOSE_PROJECT_NAME=ushadow + +# ========================================== +# PORT CONFIGURATION +# ========================================== +PORT_OFFSET=10 +BACKEND_PORT=8010 +WEBUI_PORT=3010 + +# ========================================== +# DATABASE ISOLATION +# ========================================== +MONGODB_DATABASE=ushadow +REDIS_DATABASE=0 + +# ========================================== +# CORS & FRONTEND CONFIGURATION +# ========================================== +CORS_ORIGINS=http://localhost:3010,http://127.0.0.1:3010,http://localhost:8010,http://127.0.0.1:8010 +VITE_BACKEND_URL=http://localhost:8010 +VITE_ENV_NAME=ushadow +HOST_IP=localhost + +# Development mode +DEV_MODE=true + +# ========================================== +# SHARE LINK CONFIGURATION +# ========================================== +# Base URL for share links (highest priority if set) +# SHARE_BASE_URL=https://ushadow.tail12345.ts.net + +# Public gateway URL for external friend sharing (requires share-gateway deployment) +# SHARE_PUBLIC_GATEWAY=https://share.yourdomain.com + +# Share feature toggles +SHARE_VALIDATE_RESOURCES=false # Enable strict resource validation +SHARE_VALIDATE_TAILSCALE=false # Enable Tailscale IP validation + +# ========================================== +# KEYCLOAK CONFIGURATION +# ========================================== +# SECURITY: Change these defaults in production! +KEYCLOAK_ADMIN=admin +KEYCLOAK_ADMIN_PASSWORD=changeme +KEYCLOAK_PORT=8081 +KEYCLOAK_MGMT_PORT=9000 diff --git a/.githooks/README.md b/.githooks/README.md new file mode 100644 index 00000000..d7e000fb --- /dev/null +++ b/.githooks/README.md @@ -0,0 +1,49 @@ +# Git Hooks + +This directory contains git hooks that are **committed to the repository**. + +## Setup (One-Time) + +After cloning, configure git to use these hooks: + +```bash +git config core.hooksPath .githooks +``` + +## Automatic Setup + +Add this to your `~/.gitconfig` to automatically use `.githooks` in all repos: + +```ini +[init] + templateDir = ~/.git-templates +``` + +Then create `~/.git-templates/hooks/post-clone`: +```bash +#!/bin/bash +if [ -d .githooks ]; then + git config core.hooksPath .githooks +fi +``` + +## Available Hooks + +### post-checkout +Automatically configures sparse checkout for chronicle and mycelia submodules to prevent circular dependencies. + +**What it does:** +- Chronicle: Excludes `extras/mycelia/` +- Mycelia: Excludes `friend/` + +**When it runs:** +- After `git checkout` +- After `git submodule update` +- After initial clone (with setup) + +## Testing + +Test the hook manually: +```bash +./.githooks/post-checkout +``` diff --git a/.githooks/post-checkout b/.githooks/post-checkout new file mode 100755 index 00000000..5537edff --- /dev/null +++ b/.githooks/post-checkout @@ -0,0 +1,45 @@ +#!/bin/bash +# Post-checkout hook to configure sparse checkout for submodules +# This prevents circular dependencies between chronicle and mycelia + +set -e + +echo "๐ง Configuring sparse checkout for submodules..." + +# Configure chronicle to exclude extras/mycelia +if [ -d "chronicle" ]; then + CHRONICLE_GIT="$(cd chronicle && git rev-parse --git-dir 2>/dev/null || echo "")" + if [ -n "$CHRONICLE_GIT" ] && [ -d "$CHRONICLE_GIT" ]; then + echo " ๐ Configuring chronicle (excluding extras/mycelia)" + mkdir -p "$CHRONICLE_GIT/info" + cat > "$CHRONICLE_GIT/info/sparse-checkout" <<'SPARSE' +/* +!extras/mycelia/ +SPARSE + (cd chronicle && git config core.sparseCheckout true && git read-tree -mu HEAD 2>/dev/null || true) + fi +fi + +# Configure mycelia to exclude friend +if [ -d "mycelia" ]; then + MYCELIA_GIT="$(cd mycelia && git rev-parse --git-dir 2>/dev/null || echo "")" + if [ -n "$MYCELIA_GIT" ] && [ -d "$MYCELIA_GIT" ]; then + echo " ๐ Configuring mycelia (excluding friend)" + mkdir -p "$MYCELIA_GIT/info" + cat > "$MYCELIA_GIT/info/sparse-checkout" <<'SPARSE' +/* +!friend/ +SPARSE + (cd mycelia && git config core.sparseCheckout true && git read-tree -mu HEAD 2>/dev/null || true) + fi +fi + +# Configure openmemory (no exclusions needed currently) +if [ -d "openmemory" ]; then + OPENMEMORY_GIT="$(cd openmemory && git rev-parse --git-dir 2>/dev/null || echo "")" + if [ -n "$OPENMEMORY_GIT" ] && [ -d "$OPENMEMORY_GIT" ]; then + echo " ๐ Openmemory configured (no exclusions)" + fi +fi + +echo "โ Sparse checkout configured successfully" diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 00000000..2bc78456 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,12 @@ +[submodule "chronicle"] + path = chronicle + url = https://github.com/Ushadow-io/chronicle.git + update = checkout + +[submodule "mycelia"] + path = mycelia + url = https://github.com/mycelia-tech/mycelia.git + update = checkout +[submodule "openmemory"] + path = openmemory + url = https://github.com/Ushadow-io/mem0.git diff --git a/DECISION_POINT_1.md b/DECISION_POINT_1.md new file mode 100644 index 00000000..a4c773b6 --- /dev/null +++ b/DECISION_POINT_1.md @@ -0,0 +1,193 @@ +# Decision Point #1: Resource Validation + +## Current Status + +โ **Feature is ready to use** - Sharing works with lazy validation (no resource checking) +๐ **Your choice** - Implement strict validation if you want to prevent broken share links + +## How It Works Now + +When you create a share link, the system: +1. โ Creates share token in MongoDB +2. โ Generates share URL +3. โ ๏ธ **Does NOT verify** the conversation/resource exists +4. โ Returns link to user + +**Result**: Share links are created instantly, but might be broken if the resource doesn't exist. + +--- + +## Enabling Strict Validation + +### Step 1: Set Environment Variable + +Add to your `.env` file: +```bash +SHARE_VALIDATE_RESOURCES=true +``` + +This tells the share service to validate resources before creating share links. + +### Step 2: Implement Validation Logic + +**Location**: `ushadow/backend/src/services/share_service.py` line ~340 + +I've prepared the function structure. You need to add **5-10 lines** of code to validate the resource exists. + +--- + +## Implementation Options + +Since Mycelia uses a resource-based API (not REST), you have two approaches: + +### Option A: Validate via Mycelia Objects API (Recommended) + +```python +# In _validate_resource_exists(), around line 340: + +if resource_type == ResourceType.CONVERSATION: + try: + async with httpx.AsyncClient(timeout=5.0) as client: + # Call Mycelia objects resource with "get" action + response = await client.post( + "http://mycelia-backend:8000/api/resource/tech.mycelia.objects", + json={ + "action": "get", + "id": resource_id + }, + headers={"Authorization": f"Bearer {self._get_service_token()}"} + ) + + if response.status_code == 404: + raise ValueError(f"Conversation {resource_id} not found in Mycelia") + elif response.status_code != 200: + raise ValueError(f"Failed to validate conversation: {response.status_code}") + + except httpx.RequestError as e: + logger.error(f"Failed to connect to Mycelia: {e}") + raise ValueError("Could not connect to Mycelia to validate conversation") +``` + +**Pros**: Validates against Mycelia directly +**Cons**: Requires service token for authentication + +--- + +### Option B: Validate via Ushadow Generic Proxy + +```python +if resource_type == ResourceType.CONVERSATION: + try: + async with httpx.AsyncClient(timeout=5.0) as client: + # Use ushadow's generic proxy to Mycelia + response = await client.post( + "http://localhost:8080/api/services/mycelia-backend/proxy/api/resource/tech.mycelia.objects", + json={ + "action": "get", + "id": resource_id + } + ) + + if response.status_code == 404: + raise ValueError(f"Conversation {resource_id} not found") + elif response.status_code != 200: + raise ValueError(f"Failed to validate conversation: {response.status_code}") + + except httpx.RequestError as e: + logger.error(f"Mycelia validation failed: {e}") + raise ValueError("Could not validate conversation") +``` + +**Pros**: Leverages existing proxy, handles auth automatically +**Cons**: Assumes ushadow proxy is available + +--- + +### Option C: Skip Validation (Current Behavior) + +Don't set `SHARE_VALIDATE_RESOURCES=true` and leave the TODO as-is. + +**Pros**: Instant share creation, no API calls +**Cons**: Users might create broken share links + +--- + +## Trade-offs to Consider + +| Aspect | Lazy Validation | Strict Validation | +|--------|----------------|-------------------| +| **Speed** | โ Instant (~5ms) | โ ๏ธ Slower (~50-100ms) | +| **Reliability** | โ ๏ธ Might create broken links | โ Only valid links | +| **UX** | โ Fast feedback | โ ๏ธ Slight delay | +| **Dependencies** | โ No backend calls | โ ๏ธ Requires Mycelia/Chronicle | +| **Error handling** | โ ๏ธ Broken links fail silently | โ Immediate error feedback | + +--- + +## My Recommendation + +**Start with Lazy Validation (current behavior)** because: +1. It's simpler - no extra code needed +2. Users rarely share non-existent conversations +3. When they access a broken link, they get a clear "not found" error +4. You can always add strict validation later if needed + +**Implement Strict Validation if:** +- You have frequent issues with broken share links +- You want immediate feedback during share creation +- The ~50-100ms delay is acceptable for your UX + +--- + +## Testing Your Implementation + +Once you've implemented validation: + +```bash +# Test with valid conversation +curl -X POST http://localhost:8080/api/share/create \ + -H "Content-Type: application/json" \ + -H "Cookie: ushadow_auth=YOUR_TOKEN" \ + -d '{ + "resource_type": "conversation", + "resource_id": "VALID_CONVERSATION_ID", + "permissions": ["read"] + }' + +# Expected: 201 Created with share URL + +# Test with invalid conversation +curl -X POST http://localhost:8080/api/share/create \ + -H "Content-Type: application/json" \ + -H "Cookie: ushadow_auth=YOUR_TOKEN" \ + -d '{ + "resource_type": "conversation", + "resource_id": "INVALID_ID_12345", + "permissions": ["read"] + }' + +# Expected: 400 Bad Request with "Conversation not found" error +``` + +--- + +## Questions? + +**Q: What if Mycelia/Chronicle is down during validation?** +A: The validation will fail with "Could not connect" error, preventing share creation. Consider adding retry logic or circuit breaker. + +**Q: Should I validate memories too?** +A: Yes, add similar logic for `ResourceType.MEMORY` if users can share individual memories. + +**Q: Can I validate asynchronously (background job)?** +A: Not recommended - user needs immediate feedback. If validation is slow, consider caching resource existence. + +--- + +## Next Steps + +1. **Decide**: Lazy vs Strict validation +2. **If Strict**: Set `SHARE_VALIDATE_RESOURCES=true` in `.env` +3. **Implement**: Add 5-10 lines in `share_service.py` (see options above) +4. **Test**: Create shares with valid/invalid IDs +5. **Move to Decision Point #2**: User authorization checks diff --git a/DECISION_POINT_3.md b/DECISION_POINT_3.md new file mode 100644 index 00000000..1c9531ff --- /dev/null +++ b/DECISION_POINT_3.md @@ -0,0 +1,186 @@ +# Decision Point #3: Tailscale Network Validation + +## Current Status + +โ **Feature is optional** - Tailscale validation only applies when users create shares with `tailscale_only=true` +๐ **Your choice** - Implement if you want to restrict certain shares to your Tailscale network + +## How It Works Now + +**Without Tailscale validation** (current default): +- `tailscale_only=false` shares โ Accessible from anywhere โ +- `tailscale_only=true` shares โ Still accessible from anywhere โ ๏ธ (validation disabled) + +**With Tailscale validation** (when implemented): +- `tailscale_only=false` shares โ Accessible from anywhere โ +- `tailscale_only=true` shares โ Only accessible from Tailnet โ (validated) + +--- + +## When Do You Need This? + +**Skip Tailscale validation if:** +- You only use the public share gateway (all shares are `tailscale_only=false`) +- You trust users not to abuse `tailscale_only` flag +- Simpler setup is more important than this specific security control + +**Implement Tailscale validation if:** +- You want users to create Tailnet-only shares (private conversations) +- You expose ushadow directly to your Tailnet (not just via gateway) +- You need strong network-based access control + +--- + +## Implementation Options + +### Option A: IP Range Check (Recommended for Direct Tailscale) + +If ushadow runs **directly as a Tailscale node** (not behind a proxy): + +```python +# In share_service.py:_validate_tailscale_access(), around line 465: + +try: + ip = ipaddress.ip_address(request_ip) + tailscale_range = ipaddress.ip_network("100.64.0.0/10") + is_tailscale = ip in tailscale_range + logger.debug(f"IP {request_ip} {'is' if is_tailscale else 'is NOT'} in Tailscale range") + return is_tailscale +except ValueError: + logger.warning(f"Invalid IP address: {request_ip}") + return False +``` + +**How it works**: +- Tailscale uses CGNAT IP range 100.64.0.0/10 +- Check if request IP falls in this range +- Fast, no API calls + +**Pros**: Simple, fast, no external dependencies +**Cons**: Only works if ushadow is directly on Tailscale (not behind nginx/proxy) + +**Enable**: Set `SHARE_VALIDATE_TAILSCALE=true` in `.env` + +--- + +### Option B: Tailscale Serve Headers (For Tailscale Serve Setup) + +If you expose ushadow via **Tailscale Serve** (reverse proxy): + +**Current limitation**: This requires passing the full `Request` object, not just IP. + +**Architecture change needed**: +```python +# In share_service.py:validate_share_access() +# Instead of: +is_tailscale = await self._validate_tailscale_access(request_ip) + +# Pass full request: +is_tailscale = await self._validate_tailscale_access(request) + +# In _validate_tailscale_access(): +async def _validate_tailscale_access(self, request: Request) -> bool: + tailscale_user = request.headers.get("X-Tailscale-User") + if tailscale_user: + logger.debug(f"Validated Tailscale user: {tailscale_user}") + return True + return False +``` + +**How it works**: +- Tailscale Serve adds `X-Tailscale-User` header with authenticated user +- If header present โ user is on your Tailnet +- Cryptographically verified by Tailscale + +**Pros**: Most secure, user identity available +**Cons**: Requires refactoring to pass Request object, only works with Tailscale Serve + +--- + +### Option C: Skip Validation (Current Default) + +Don't set `SHARE_VALIDATE_TAILSCALE=true` and leave as-is. + +**What happens**: +- All shares work regardless of IP +- `tailscale_only` flag is ignored (becomes cosmetic) +- Simpler setup, no code changes needed + +**Trade-off**: Users can't create truly Tailnet-restricted shares + +--- + +## My Recommendation + +### For Your Use Case (Public Gateway Architecture): + +**Skip Tailscale validation for now** because: + +1. **Your architecture**: Friends access via public gateway, not directly to ushadow +2. **Gateway handles it**: The gateway itself is on your Tailnet, providing network isolation +3. **Simpler**: One less thing to configure and maintain +4. **The flag still useful**: Even without validation, `tailscale_only` serves as metadata/intent + +**When you WOULD need it**: +- If users access ushadow directly via Tailscale (not just gateway) +- If you want to enforce Tailnet-only shares for specific conversations + +--- + +## Architecture Reminder + +``` +Public Share (tailscale_only=false): +Friend โ Public Gateway โ [Tailscale] โ ushadow + +Tailscale-Only Share (tailscale_only=true): +Friend on your Tailnet โ ushadow (direct access) + โ THIS is where Tailscale validation matters +``` + +The validation prevents a friend from accessing a `tailscale_only` share via the public gateway or from outside your network. + +--- + +## Testing Your Implementation + +Once implemented: + +```bash +# 1. Create Tailscale-only share +curl -X POST http://localhost:8080/api/share/create \ + -H "Content-Type: application/json" \ + -H "Cookie: ushadow_auth=YOUR_TOKEN" \ + -d '{ + "resource_type": "conversation", + "resource_id": "abc123", + "permissions": ["read"], + "tailscale_only": true + }' + +# 2. Try to access from Tailscale IP (100.64.x.x) +# Expected: โ Access granted + +# 3. Try to access from public IP (not Tailscale) +# Expected: โ 403 "Access restricted to Tailscale network" +``` + +--- + +## Summary + +| Option | When to Use | Complexity | Security | +|--------|-------------|------------|----------| +| **Skip** | Public gateway only | โญ Easy | Medium (gateway isolated) | +| **IP Range** | Direct Tailscale access | โญโญ Medium | High (network-level) | +| **Serve Headers** | Tailscale Serve setup | โญโญโญ Complex | Highest (crypto verified) | + +**Recommended**: Skip for now, implement later if needed. + +--- + +## Next Steps + +1. **Decide**: Do you need Tailscale-only shares? +2. **If No**: Leave as-is, move to frontend integration +3. **If Yes**: Set `SHARE_VALIDATE_TAILSCALE=true` and add 5-10 lines (Option A) diff --git a/KEYCLOAK_LOGIN_FIXES.md b/KEYCLOAK_LOGIN_FIXES.md new file mode 100644 index 00000000..4ede3654 --- /dev/null +++ b/KEYCLOAK_LOGIN_FIXES.md @@ -0,0 +1,296 @@ +# Keycloak Login Fixes - Complete Summary + +## Issues Fixed + +### 1. โ Login Page Shows Password Fields +**Problem**: Login page displayed username/password fields, but these were non-functional (Keycloak handles credentials, not the app). + +**Solution**: Replaced the entire login form with a single "Sign in with Keycloak" button that clearly indicates users will be redirected to Keycloak for authentication. + +**File Changed**: `ushadow/frontend/src/pages/LoginPage.tsx` + +**Before**: +```tsx + + + +``` + +**After**: +```tsx + +
You'll be redirected to Keycloak for secure authentication
+``` + +--- + +### 2. โ OAuth Callback Route Missing +**Problem**: After successful Keycloak login, users were redirected to `/oauth/callback?code=...`, but this route wasn't registered in the app. React Router didn't recognize it, so it redirected to the login page, creating an infinite loop. + +**Solution**: Added the OAuth callback route as a public route in App.tsx. + +**File Changed**: `ushadow/frontend/src/App.tsx` + +**Changes**: +1. Imported OAuthCallback component: + ```tsx + import OAuthCallback from './auth/OAuthCallback' + ``` + +2. Registered the route: + ```tsx + {/* Public Routes */} +{error}
+ +Completing sign-in...
+
+ {ingressHostname}
+
+
+ + Accessible at: http://{ingressHostname} +
+