📄 Smart Document Q&A – Technical Overview & Progress Log

✅ Objective

To build a real-world RAG (Retrieval-Augmented Generation) system that can:

• Index a collection of PDF documents

• Allow users to ask natural language questions

• Retrieve relevant context and generate coherent, grounded answers using an LLM

🏗️ System Architecture

1. Backend (FastAPI)

• Accepts PDF uploads via /upload

• Extracts and chunks text into smaller segments

• Embeds each chunk using all-MiniLM-L6-v2 from sentence-transformers

• Stores vectors in a FAISS index (inner product for cosine similarity)

• Metadata (chunk, source filename, etc.) saved in JSON

• /ask endpoint performs:

  • Query embedding

  • FAISS retrieval of top-k chunks

  • Prompt construction with context

  • LLM call via local llama-simple binary (gguf model)

  • Return of answer

2. Frontend (Next.js + Tailwind)

• Simple interface for:

  • Typing a question

  • Viewing a loading state

  • Displaying final answers and sources

• Calls /ask endpoint from backend via lib/api.ts

🧠 What Works Well

• ✅ Document ingestion works for PDFs.

• ✅ Chunking and embedding pipeline is functional.

• ✅ FAISS retrieval gives relevant chunks.

• ✅ llama-simple can generate responses based on prompt.

• ✅ Frontend successfully queries backend and shows answers.

⚠️ Issues & Limitations

• 🔸 Prompt sometimes misleads the LLM into hallucinations.

• 🔸 Answers were occasionally not grounded in retrieved content.

• 🔸 LLM was not reliably using source context until we structured the prompt better.

• 🔸 The local model (llama-simple) isn't fine-tuned for QA.

• 🔸 No reranking or source-based highlighting yet.

• 🔸 Some responses were poorly formatted (e.g., odd punctuation, clipped endings).

🔧 Recent Improvements

• ✅ Step-by-step RAG architecture implemented

• ✅ Added prompt engineering to improve grounding

• ✅ Began parsing structured output (JSON-formatted answers)

• ✅ Embedded chunk metadata into prompts for explainability

• ✅ Clear separation of backend (retrieval & LLM) vs frontend

📦 Data & Models Used

• Model: all-MiniLM-L6-v2 (for embeddings)

• LLM: Quantized LLaMA-2 7B (via llama.cpp)

• Indexing: FAISS (flat inner product)

• Docs: Local PDF files (manually uploaded)

📅 Next Steps (Planned)

1. **Reranking using BAAI/bge-m3 or Cohere Reranker
   **

2. Better structured output from LLM (JSON + citations)

3. Citation highlighting on frontend (clickable chunks)

4. Multi-doc support UI (PDF-specific filtering)

5. Model fallback or hybrid LLM options (e.g., OpenAI for better answers)

6. Upload interface & file explorer