What Is RAG? Let AI Read Your Docs Before Answering

LLM knowledge stops at the training cutoff and excludes your internal docs. RAG (Retrieval-Augmented Generation) fixes that: search relevant documents first, then answer from them.

Why RAG

User: "What is our company's PTO policy?"

Without RAG → model invents a plausible policy
With RAG    → retrieve employee handbook → answer from real docs → cite sources

Three typical scenarios:

• Enterprise Q&A: policies, processes, product docs
• Support bots: FAQs and manuals
• Code assistants: your repo (Cursor does this)

Full Pipeline (Four Steps)

         Offline (prepare ahead)              Online (at query time)
┌─────────────────────────┐          ┌─────────────────────────┐
│  1. Chunk documents      │          │  4. User asks question   │
│  2. Embed chunks         │          │     ↓                   │
│  3. Store in vector DB   │          │  5. Embed the question  │
└─────────────────────────┘          │     ↓                   │
                                     │  6. Retrieve top chunks │
                                     │     ↓                   │
                                     │  7. Docs + question → LLM│
                                     │     ↓                   │
                                     │  8. Answer with citations│
                                     └─────────────────────────┘

Step 1: Chunking

Split long docs into segments (often 200–1000 tokens):

employee-handbook.pdf (50 pages)
  → Chunk 1: "PTO: after one year, 5 days annual leave…"
  → Chunk 2: "Sick leave: up to 1 day per month…"
  → Chunk 3: "Expense process: submit form with receipt…"

Chunk strategy affects quality—too small loses context; too large hurts precision.

Step 2: Embedding

Each chunk becomes a numeric vector via an embedding model:

"5 days PTO" → [0.12, -0.34, 0.56, ..., 0.78]  (1536 dimensions)

Semantically similar text → nearby vectors. That is how RAG matches meaning, not just keywords.

Step 3: Retrieval

The question is embedded; the database returns nearest chunks:

Question vector → nearest neighbor search → Top 3:
  1. "PTO: one year tenure, 5 days" (similarity 0.92)
  2. "Benefits overview: PTO, sick leave…" (0.78)
  3. "Leave request process…" (0.65)

Step 4: Generation

Retrieved chunks + question go to the LLM:

Answer using the materials below. If not covered, say "I'm not sure."

Materials:
[Chunk 1]
[Chunk 2]

Question: How many PTO days do I get?

Common RAG Failures

Issue	Cause	Fix
Nothing relevant retrieved	Bad chunks / wrong embedding model	Tune chunk size; swap embeddings
Right doc, low rank	Pure vector weak on exact terms	Hybrid search (vector + keyword)
Answer still wrong	Noisy chunks / too few hits	Rerank top results
Stale answers	Index not updated	Pipeline: doc change → re-index

vs. Long Context

“Context windows are huge—do we still need RAG?”

Yes:

• Cost: stuffing 100 pages per query costs far more than retrieval
• Accuracy: curated chunks often beat dumping full documents
• Freshness: update the index without retraining

Summary

RAG = retrieval + generation. Find relevant docs, then let the LLM answer from them. It remains the most practical way to use private data with AI.

Next: when AI doesn’t just talk—it acts. Agents.