🧠 MyKB — OSS Overview & Base Architecture

MyKB is an on-prem, zero-trust knowledge engine: crawl → chunk → embed → store → retrieve → compress → answer — wrapped with a hardened auth plane and an audit-friendly API surface.

It runs fully offline or with pluggable cloud LLMs, and ships in two modes:

• Solo Mode (No Auth): Local, private, developer-first. Runs on localhost with no authentication. Perfect for OSS hackers and indie devs.
• Team Mode (Auth Enabled): JWT-based authentication with short-lived tokens and refresh support. Every query is policy-filtered before retrieval. Perfect for small teams that need zero-trust enforcement.

---

1. High-Level System

+-------------------------+           +--------------------------+
|  Identity & Policy      |  JWT/SSO  |  Gateway (Planned)       |
|  (Auth Core)            +---------->+  Policy Filters + Tools  |
|  - EdDSA JWT            |           |  kb.search / kb.ingest   |
|  - Refresh tokens       |           +-----------+--------------+
|  - Device fingerprint   |                       |
|  - Admin sessions (IP)  |                       |  policy-enforced queries
+-----------+-------------+                       v
            |                          +-----------------------------+
            |                          |  KB Core (RAG Engine)       |
            |                          |  - Crawlers (site profiles) |
            |                          |  - Markdown AST chunker     |
            |                          |  - Dense + sparse embeds    |
            |                          |  - Qdrant hybrid index      |
            |                          |  - Rerank + MMR + compress  |
            |                          |  - Incremental ledger       |
            |                          +---------------+-------------+
            |                                          |
            |                              +-----------v------------+
            |                              | Vector DB (Qdrant)     |
            |                              | - named vectors        |
            |                              | - payload/metadata     |
            |                              +------------------------+
            |
            +-----> Observability & Admin:
                    - JSON logs, rate limits, health endpoints
                    - Future: Feedback → Patch Registry → Eval sets

• Auth Core = production-grade identity, tokens, sessions, rate limits, audits.
• KB Core = RAG pipeline (hybrid retrieval, reranking, context compression), ingestion, and incremental updates.
• Gateway (planned) = single entry point that enforces policy pre-retrieval (zero-trust) and exposes standardized tools.

---

2. Core Subsystems

2.1 KB Core (RAG Engine)

• Crawling & Site Profiles: Headless crawler with per-domain configs (CSS scoping, link pruning).
• Markdown AST Chunking: Token-aware, structure-preserving chunker (headings, code, tables). Stable chunk IDs + section anchors.
• Embeddings: Dense (FastEmbed/ONNX) + BM25 sparse vectors for hybrid recall.
• Hybrid Retrieval: Dense + sparse, RRF fusion, optional cross-encoder reranker, MMR for diversity.
• Context Compressor: Semantic sentence extraction to keep answers concise but grounded.
• Incremental Ledger (SQLite): Detects chunk-level changes via content hashes → upsert only deltas.
• API (FastAPI): /seed/preview, /ingest/urls, /search, /search/code_examples, /health.

2.2 Vector Store (Qdrant)

• Collections for docs + code examples.
• Named vectors for dense + sparse.
• Payload holds URL, domain, chunk_id, section path, etc.
• Bulk or incremental index modes; configurable shards.

2.3 Identity & Policy (Auth Core)

• EdDSA (Ed25519) JWT for access.
• Refresh tokens are DB-backed and revocable.
• Device fingerprint binding.
• Admin plane: session + IP whitelist.
• Rate limiting: per-route; CSP & CORS hardened.
• Audit logs: structlog JSON.

---

3. Data & Metadata

3.1 Chunk Schema (Conceptual)

• chunk_id (stable, deterministic)
• url, domain, section_path, heading
• type = text|code|table|heading
• text or content block
• embeddings: dense[], sparse{indices,values}
• metadata: neighbor windowing, compressed/original lengths

3.2 Ledger (Incremental Ingestion)

• documents: (url, doc_hash, chunk_ids, etag, last_modified)
• chunk_hashes: (url, chunk_id, chunk_hash)

This drives delta updates: add, delete, or change only what’s new.

---

4. Request Lifecycle (Search Path)

1. Client → Gateway/Auth with JWT → derive policy filter.
2. KB Core applies hybrid retrieval under filter constraints.
3. Optional rerank (cross-encoder) + MMR for diversity.
4. Neighbor expansion + semantic compression → concise context.
5. Answering: optional LLM (local or cloud) uses retrieved context + citations.

---

5. Security Model

• Auth plane: asymmetric JWTs, refresh rotation, device binding, admin sessions with IP allowlist.
• Zero-trust enforcement (planned Gateway): every query filtered pre-retrieval.
• Auditability: structured logs, health checks, (planned) per-query decision trails.

---

6. Deployment Topology

• Small: single host (Auth + KB + Qdrant) with optional GPU.
• Medium: Auth isolated; KB + Qdrant on separate node.
• Large: multi-node Qdrant (shards/replicas), autoscaled KB workers.

All containerized via Docker; config managed with .env.

---

7. Enterprise Feature Set

Already in OSS

• Crawl/ingest with site profiles
• Markdown AST chunker
• Hybrid search (dense+BM25), RRF, MMR
• Optional reranker, semantic compression
• Incremental ledger & precise chunk deletes
• Clean REST API
• Auth Core reference (JWT, refresh, device bind, rate limits, CSP/CORS)

Planned Enterprise

• Gateway with policy-aware pre-retrieval filters
• Multi-tenancy isolation (collections or payload encryption)
• Policy-aware chunking (per-section ACLs)
• Feedback → Patch Registry → Eval Sets (self-healing loop)
• Solution Packs (domain-specific tuning)
• Lineage graph across sources (code/docs/tickets)
• Compliance Co-Pilot (policy enforcement on code/docs)

---

8. Non-Functional Guarantees

• On-prem first: fully offline path.
• Performance: small-model embeddings, GPU optional.
• Reliability: idempotent ingestion, ledgered changes.
• Portability: Python, FastAPI, ONNX/FastEmbed, Qdrant, Docker.

---

9. Services & Data (What Runs Where)

Services:

• KB Core API (FastAPI)
• Auth Core API (FastAPI)
• Qdrant vector store

Data:

• Vector DB: embeddings + metadata
• Ledger (SQLite): ingestion state
• Auth DB (Postgres): users, tokens, devices

Ops:

• Health endpoints, JSON logs, rate limits, CSP/CORS, Docker.

---

10. Roadmap (90 Days)

• M1 Foundations: Gateway MVP (JWT → filter map → pre-retrieval); OSS "Team" RBAC.
• M2 Safety + Packs: Feedback UI → Patch Registry + Eval harness; release Solution Packs.
• M3 Enterprise: Policy-aware chunking; multi-tenant isolation; lineage prototype; SSO/SAML/OIDC.

---

11. What’s Not in This Document

• No repo URLs, tokens, or secrets.
• No IPs, keys, or deployment envs.
• No customer data models.

---

👉 This README is an overview & architecture document for MyKB OSS. Code will follow; for now, this sets the stage for collaborators, testers, and enterprises to see what we’re building.