diff --git a/.claude/plugins/test-automation/plugin.json b/.claude/plugins/test-automation/plugin.json index 50c9a0b5..4a445fe0 100644 --- a/.claude/plugins/test-automation/plugin.json +++ b/.claude/plugins/test-automation/plugin.json @@ -10,6 +10,8 @@ "skills": [ "skills/spec.md", "skills/qa-test-cases.md", - "skills/automate-tests.md" + "skills/automate-tests.md", + "skills/debug-robot-browser.md", + "skills/debug-robot-api.md" ] } diff --git a/.claude/plugins/test-automation/skills/debug-robot-api.md b/.claude/plugins/test-automation/skills/debug-robot-api.md new file mode 100644 index 00000000..7d8acab5 --- /dev/null +++ b/.claude/plugins/test-automation/skills/debug-robot-api.md @@ -0,0 +1,323 @@ +--- +name: debug-robot-api +description: This skill should be used when the user asks to "debug robot api tests", "fix a failing api test", "diagnose a robot api test failure", "why is my robot api test failing", or "run api tests and fix them". Drives a fast run-parse-fix cycle for Robot Framework API tests using RequestsLibrary. +--- + +## Purpose + +Drive a fast debug cycle for failing Robot Framework API tests: +**run with debug logging → parse output for HTTP details → identify cause → fix → verify**. + +API tests fail at the HTTP layer (status codes, response bodies, auth tokens) — not the DOM. The debug workflow is entirely different from browser tests. + +--- + +## Step 1 — Run with debug logging + +```bash +# All API tests with verbose HTTP output +cd robot_tests && pixi run robot --loglevel DEBUG api/ + +# Single suite (fastest) +cd robot_tests && pixi run robot --loglevel DEBUG api/keycloak_auth.robot + +# Single test by name +cd robot_tests && pixi run robot --loglevel DEBUG --test "TC-KC-001*" api/keycloak_auth.robot +``` + +`--loglevel DEBUG` logs the full request URL, headers, and response body for every HTTP call. Without it, you only see the status code. + +--- + +## Step 2 — Parse the failure output + +### output.xml — status codes and response bodies + +```bash +# All FAIL lines with surrounding context +python3 -c " +import xml.etree.ElementTree as ET +try: + tree = ET.parse('robot_tests/output.xml') + for msg in tree.getroot().iter('msg'): + if msg.get('level') == 'FAIL' or (msg.text and 'status' in (msg.text or '').lower()): + print(msg.text[:300]) + print('---') +except: pass +" 2>/dev/null | head -80 + +# Quick grep for HTTP status codes in output +grep -o 'status.*[0-9]\{3\}\|[0-9]\{3\}.*status\|HTTPError\|ConnectionError\|FAIL' \ + robot_tests/output.xml | sort -u | head -20 + +# Response body at failure point +grep -o '.\{0,50\}response\|body\|detail.\{0,200\}' robot_tests/output.xml | head -20 +``` + +### log.html — easiest to read + +Open in a browser for full formatted output with collapsible sections: +```bash +open robot_tests/log.html +``` + +### Backend container logs + +Often the clearest signal — shows the actual exception or validation error: +```bash +docker logs ushadow-test-backend-test-1 --tail 50 + +# Follow while running tests +docker logs ushadow-test-backend-test-1 -f +``` + +--- + +## Step 3 — Identify the failure pattern + +### Pattern A: Connection refused / backend not ready + +**Symptom:** +``` +ConnectionError: HTTPConnectionPool(host='localhost', port=8200): Max retries exceeded +``` + +**Cause:** Backend container not running or still starting. + +**Diagnosis:** +```bash +# Check container status +docker ps | grep ushadow-test + +# Check if backend responds +curl -s http://localhost:8200/health | python3 -m json.tool + +# Start containers if needed +cd robot_tests && make start +``` + +**Fix:** Wait for containers before running. The suite setup handles this — if containers are already running it reuses them. If not, run `make start` first or ensure the suite setup keyword is included. + +--- + +### Pattern B: 401 Unauthorized + +**Symptom:** +``` +Expected status 200 but got 401 +``` + +**Cause:** Missing auth token, expired token, or wrong client credentials. + +**Diagnosis:** +```bash +# Check if token endpoint works +curl -s -X POST http://localhost:8181/realms/ushadow/protocol/openid-connect/token \ + -d "grant_type=password&client_id=ushadow-frontend&username=kctest@example.com&password=TestKeycloak123!" \ + | python3 -m json.tool + +# Check if test user exists in Keycloak +TOKEN=$(curl -s -X POST http://localhost:8181/realms/master/protocol/openid-connect/token \ + -d "grant_type=password&client_id=admin-cli&username=admin&password=admin" \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])") +curl -s -H "Authorization: Bearer $TOKEN" \ + "http://localhost:8181/admin/realms/ushadow/users?email=kctest@example.com" | python3 -m json.tool +``` + +**Common fixes:** +- Test user not created → suite setup should call `Ensure Keycloak Test User Exists` +- Wrong client_id in token request → check `KEYCLOAK_CLIENT_ID` in `test_env.py` (should be `ushadow-frontend` or `ushadow-cli`) +- Token expired mid-suite → refresh in suite setup or request a fresh token per test + +--- + +### Pattern C: 403 Forbidden + +**Symptom:** +``` +Expected status 200 but got 403 +``` + +**Cause:** Token present but insufficient permissions, or using wrong client type. + +**Common case — introspection with public client:** +Keycloak's token introspection endpoint (`/protocol/openid-connect/token/introspect`) requires a **confidential client** with client credentials. The `ushadow-frontend` client is public and will get 401/403. + +**Fix:** +```robot +# Check status gracefully — skip if introspection not supported +${response}= POST On Session ... expected_status=any +IF '${response.status_code}' in ['401', '403'] + Skip Token introspection requires confidential client (public client limitation) +END +Should Be Equal As Integers ${response.status_code} 200 +``` + +Or use the CLI client (`ushadow-cli`) if it's configured as confidential: +```robot +# Use CLI client for introspection +${token}= Get Token Via CLI Client +``` + +--- + +### Pattern D: 404 Not Found + +**Symptom:** +``` +Expected status 200 but got 404 +``` + +**Cause:** Endpoint path is wrong, or the route doesn't exist. + +**Diagnosis:** +```bash +# Check what routes are registered on the backend +curl -s http://localhost:8200/openapi.json | python3 -c " +import sys, json +spec = json.load(sys.stdin) +for path in sorted(spec.get('paths', {}).keys()): + print(path) +" | grep -i auth + +# Or view Swagger UI +open http://localhost:8200/docs +``` + +**Fix:** Correct the endpoint path in the robot file. Check the backend's router to confirm the exact path. + +--- + +### Pattern E: Response body mismatch + +**Symptom:** +``` +'expected_field' != 'actual_field' +Should Be Equal As Strings FAIL +``` + +**Cause:** The response JSON structure changed, or the test is checking the wrong key. + +**Diagnosis:** +```bash +# Make the call manually and inspect the actual response +curl -s -X POST http://localhost:8200/api/auth/token \ + -H "Content-Type: application/json" \ + -d '{"code":"test","code_verifier":"test","redirect_uri":"http://localhost:3001/oauth/callback"}' \ + | python3 -m json.tool +``` + +Check the actual field names in the response vs what the test asserts. Common mismatches: +- `access_token` vs `token` +- `user_id` vs `id` +- Nested vs flat structure (`user.email` vs `email`) + +--- + +### Pattern F: Suite setup failure + +**Symptom:** Tests show as `NOT RUN` or the suite itself fails before any tests execute. + +**Diagnosis:** +```bash +# Check the first FAIL message +grep -m5 'FAIL\|ERROR' robot_tests/output.xml + +# Check container health +docker ps --format "table {{.Names}}\t{{.Status}}" | grep ushadow-test +``` + +**Common causes:** +- MongoDB not cleared (`✓ MongoDB test database cleared` should appear in console) +- Keycloak not reachable at `http://localhost:8181` +- Backend health check failing at `http://localhost:8200/health` + +--- + +## Step 4 — Common fixes reference + +### Auth keywords (from `resources/auth_keywords.robot`) + +```robot +# Get a token for the test user +${TOKEN}= Get Keycloak Token ${KEYCLOAK_TEST_EMAIL} ${KEYCLOAK_TEST_PASSWORD} + +# Ensure the test user exists (idempotent — safe to call in setup) +Ensure Keycloak Test User Exists + +# Create auth header dict +${HEADERS}= Create Dictionary Authorization=Bearer ${TOKEN} +``` + +### Session management + +```robot +# Create a session pointing at the backend +Create Session api ${BACKEND_URL} headers=${HEADERS} + +# Make a call +${resp}= GET On Session api /api/auth/me expected_status=200 + +# Log full response for debugging +Log Response: ${resp.status_code} ${resp.text} +``` + +### Graceful skip vs fail + +```robot +# Skip if feature not available (e.g. confidential client not configured) +${resp}= POST On Session api ${ENDPOINT} expected_status=any +Skip If '${resp.status_code}' == '501' Feature not implemented + +# Assert with helpful message +Should Be Equal As Integers ${resp.status_code} 200 +... Expected 200 but got ${resp.status_code}: ${resp.text} +``` + +--- + +## Step 5 — Fix and verify + +```bash +# Re-run with debug logging to confirm fix +cd robot_tests && pixi run robot --loglevel DEBUG api/keycloak_auth.robot +``` + +Expected output when passing: +``` +TC-KC-001: Authenticate with Keycloak Direct Grant | PASS | +TC-KC-002: OAuth Authorization Code Flow (skip) | SKIP | +TC-KC-003: Validate Access Token | PASS | +TC-KC-004: Introspect Access Token (skip) | SKIP | +TC-KC-005: Refresh Access Token | PASS | +TC-KC-006: Logout and Revoke Tokens | PASS | +6 tests, 4 passed, 0 failed, 2 skipped +``` + +--- + +## Environment quick reference + +| Service | Port | URL | +|---------|------|-----| +| Backend | 8200 | `http://localhost:8200` | +| Keycloak (test) | 8181 | `http://localhost:8181` | +| MongoDB (test) | 27118 | `mongodb://localhost:27118` | +| Redis (test) | 6480 | `redis://localhost:6480` | + +Keycloak admin: `admin` / `admin` +Test user: `kctest@example.com` / `TestKeycloak123!` +Realm: `ushadow` +Frontend client: `ushadow-frontend` (public) +CLI client: `ushadow-cli` (confidential, for direct grant) + +## Key files + +| File | Purpose | +|------|---------| +| `robot_tests/api/keycloak_auth.robot` | Keycloak auth test suite | +| `robot_tests/resources/auth_keywords.robot` | Shared auth keywords | +| `robot_tests/resources/setup/test_env.py` | URLs, ports, credentials | +| `robot_tests/resources/setup/suite_setup.robot` | Container start/stop | +| `robot_tests/output.xml` | Last run results | +| `robot_tests/log.html` | Human-readable log (open in browser) | diff --git a/.claude/plugins/test-automation/skills/debug-robot-browser.md b/.claude/plugins/test-automation/skills/debug-robot-browser.md new file mode 100644 index 00000000..5371bf40 --- /dev/null +++ b/.claude/plugins/test-automation/skills/debug-robot-browser.md @@ -0,0 +1,243 @@ +--- +name: debug-robot-browser +description: This skill should be used when the user asks to "debug robot browser tests", "fix a failing browser test", "diagnose a robot test failure", "why is my robot test failing", or "run browser tests and fix them". Drives a fast run-parse-fix cycle for Robot Framework browser tests (RF Browser / Playwright wrapper). +--- + +## Purpose + +Drive a fast debug cycle for failing Robot Framework browser tests: +**run with short timeout → parse failure output → identify cause → fix → verify**. + +Always prefer the reduced-timeout run for diagnosis. The default 15 s timeout means a single failing test wastes 15 s; 5 s surfaces the same failure 3× faster. + +--- + +## Step 1 — Run with reduced timeout + +```bash +# All browser tests (fastest diagnosis) +cd robot_tests && pixi run robot -v HEADLESS:true -v TIMEOUT:5s browser/ + +# Single test by name pattern (even faster) +cd robot_tests && pixi run robot -v HEADLESS:true -v TIMEOUT:5s --test "TC-BR-KC-002*" browser/ +``` + +| Variable | Diagnosis | Normal CI | +|----------|-----------|-----------| +| `TIMEOUT` | `5s` | `15s` (default in robot file) | +| `HEADLESS` | `true` | `true` | + +Do **not** edit the robot file to change the timeout — pass it via `-v`. + +--- + +## Step 2 — Parse the failure output + +After the run, three files contain everything needed: + +### output.xml — what failed and where + +```bash +# Failing URL at the moment of failure +grep -o '.\{0,80\}Failed at URL.\{0,120\}' robot_tests/output.xml + +# Expected selector that timed out +grep -o 'waiting for locator.*visible' robot_tests/output.xml | sort -u + +# Full test status summary +grep 'status status=' robot_tests/output.xml | grep FAIL +``` + +### playwright-log.txt — page source at failure time + +The log contains the full page HTML right after every screenshot. Extract it: + +```bash +# All Keycloak element IDs and classes (kc-*) +grep -o 'kc-[a-zA-Z-]*' robot_tests/playwright-log.txt | sort -u + +# All data-testid values present on the page +grep -o 'data-testid="[^"]*"' robot_tests/playwright-log.txt | sort -u + +# Full page source blob (first 4000 chars) +grep -o '"msg":"[^"]*"' robot_tests/playwright-log.txt | head -c 4000 + +# URL the browser was at when it failed +grep 'Failed at URL\|navigate\|goto' robot_tests/playwright-log.txt | grep localhost | tail -20 +``` + +### browser/screenshot/ — visual snapshot at failure + +```bash +ls -lt robot_tests/browser/screenshot/*.png | head -5 +# Open the most recent: open robot_tests/browser/screenshot/-failure.png +``` + +--- + +## Step 3 — Identify the failure pattern + +### Pattern A: "Invalid parameter: redirect_uri" on Keycloak page + +**Symptom:** Browser lands on Keycloak "We are sorry..." page with `redirect_uri` error instead of the login form. + +**Cause:** Race condition — the app's `SettingsContext` fetches the Keycloak URL from the backend (`/api/settings/config`, ~250 ms). If the login button is clicked before settings load, `keycloakConfig.url` is still the default `http://localhost:8081` (main env Keycloak). The OAuth request goes to the wrong Keycloak, which rejects the redirect URI. + +**Diagnosis:** Two Keycloak containers run simultaneously: +- `localhost:8081` — main dev environment (wrong for tests) +- `localhost:8181` — test environment (correct) + +**Fix:** Add `Wait For Load State networkidle` before clicking the login button. This waits until all in-flight network requests (including the settings API) settle. + +```robot +Navigate To Login And Wait For Settings + [Documentation] Navigate to /login and wait for settings API before clicking. + New Page ${WEB_URL}/login + Wait For Elements State css=[data-testid="login-button-keycloak"] visible timeout=${TIMEOUT} + Wait For Load State networkidle timeout=${TIMEOUT} + Click css=[data-testid="login-button-keycloak"] +``` + +Use this keyword instead of inline `New Page` + `Click` sequences. + +--- + +### Pattern B: Wrong selector for Keycloak error message + +**Symptom:** `waiting for locator('id=kc-error-message') to be visible` times out after submitting bad credentials. + +**Cause:** Keycloak 26 (PatternFly v5) moved the error from `
` to `