Embeddings and Vector Databases: The Memory Layer for AI

Every RAG system, semantic search bar, and many recommenders share one ingredient: embeddings stored in a vector database. This article explains both—accessible but complete enough to architect real systems.

The Core Idea in One Paragraph

An embedding turns text (or images, audio) into a list of numbers—a vector. Similar meaning → vectors close together in high-dimensional space. A vector database stores millions of these vectors and finds the nearest neighbors to a query in milliseconds. That is semantic search: match by meaning, not keywords.

"annual leave policy"  → [0.12, -0.34, 0.56, …]  (1536 dims)
"PTO rules"            → [0.11, -0.33, 0.55, …]  ← close!
"office parking"       → [-0.42, 0.18, 0.09, …]  ← far

How Embeddings Are Created

Embedding models (OpenAI text-embedding-3-small, Cohere, open models like bge-large) are neural networks trained so related texts land near each other:

Training intuition:
  "dog" and "puppy" → pull vectors together
  "dog" and "airplane" → push apart

Inference:
  any text → fixed-size vector (often 384–3072 dimensions)

Important properties:

Property	Why it matters
Fixed dimension	DB schema and index type depend on it
Same model for index and query	mixing models breaks similarity
Language/domain fit	legal/medical/code need appropriate models
Normalization	many DBs assume unit-length vectors for cosine distance

Distance Metrics: How “Close” Is Measured

Metric	Intuition	Common when
Cosine similarity	angle between vectors	text embeddings (default)
Euclidean (L2)	straight-line distance	some image embeddings
Dot product	magnitude + angle	normalized vectors ≈ cosine

Most text RAG uses cosine or dot product on normalized vectors.

Vector Database vs Regular Database

PostgreSQL (keyword):
  WHERE content LIKE '%PTO%'
  → misses "annual leave" unless exact words match

Vector DB:
  embed("how many vacation days?") → nearest neighbors
  → finds "annual leave policy" chunk

Vector DBs add:

• Approximate nearest neighbor (ANN) indexes—HNSW, IVF—trade tiny accuracy for huge speed
• Metadata filters—department = HR AND similarity > 0.8
• Batch upsert—re-index documents at scale
• Hybrid search—vector + keyword (critical in production)

Architecture: Where Vector DB Sits

         Ingestion (offline)                    Query (online)
┌─────────────────────────┐          ┌─────────────────────────┐
│ docs → chunk → embed    │          │ user question           │
│       → upsert vector DB│          │   → embed question      │
└─────────────────────────┘          │   → ANN search + filter │
                                     │   → top-K chunks → LLM  │
                                     └─────────────────────────┘

Same pattern powers RAG, enterprise search, and content recommendations.

Three Popular Vector DBs Compared

Pinecone — Managed, Scale-First

Type:     fully managed SaaS
Strength: zero ops, high QPS, hybrid search built-in
Weakness: cost at scale, data leaves your VPC (unless enterprise)
Best for: teams that want RAG/search live fast without running infra

Real pattern: SaaS startup indexes 500K support articles in Pinecone; p95 retrieval < 50ms; metadata filters by product line and locale.

# Conceptual flow (Pinecone SDK)
index.upsert(vectors=[{"id": "chunk-1", "values": embedding, "metadata": {"doc": "handbook"}}])
results = index.query(vector=query_embedding, top_k=5, filter={"product": "plus"})

pgvector — Postgres Extension, One Database

Type:     extension on existing PostgreSQL
Strength: ACID, joins with business data, ops team already knows Postgres
Weakness: ANN at billion-vector scale needs tuning; not a dedicated vector-only engine
Best for: < tens of millions of vectors, strong consistency, SQL workflows

Real pattern: e-commerce company stores product embeddings in products.embedding vector(1536); recommendation query joins vectors with inventory and price in one SQL transaction.

-- pgvector example
CREATE EXTENSION vector;
CREATE TABLE docs (id serial, content text, embedding vector(1536));
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops);

SELECT content FROM docs
ORDER BY embedding <=> $query_embedding
LIMIT 5;

Chroma — Developer-Friendly, Embedded or Server

Type:     open source; in-process or client/server
Strength: fast local dev, Python-first, pairs well with LangChain/LlamaIndex
Weakness: you operate scaling/HA for production server mode
Best for: prototypes, single-tenant apps, on-prem pilots

Real pattern: internal tool team embeds Chroma in a FastAPI service; engineers query internal runbooks from a Slack bot during MVP; migrate to Pinecone or pgvector when traffic grows.

Application 1: RAG (Retrieval-Augmented Generation)

Already covered in fundamentals #04—vector DB is the retrieval engine:

Shopify Sidekick-style stack:
  Help Center articles → chunks → embeddings → vector index
  Merchant question → retrieve top chunks → LLM answer + citations

Production extras beyond vanilla ANN:

• Hybrid retrieval (Shopify moved from pure vectors to vector + BM25)
• Rerankers (cross-encoder rescores top 20 → best 3)
• Freshness—re-embed on doc publish webhooks

Application 2: Semantic Search

Notion / Confluence-style "smart search":
  User: "how do I reset MFA"
  Keyword search misses "two-factor authentication recovery"
  Semantic search finds the right page

Engineering tips:

• Index title + headings + body separately with weights
• Expose snippets with similarity scores for debugging
• Log zero-result queries to expand content or tune embeddings

Application 3: Recommendations

"Similar products" / "Related articles":
  item embedding = embed(title + description + tags)
  user embedding = average of last N clicked item vectors
  recommend = nearest items excluding already seen

Spotify-style audio, Netflix thumbnails, and news feeds use the same vector similarity pattern at scale (often with specialized models, not generic text embeddings).

Choosing a Vector DB (Decision Tree)

Need managed, minimal ops, fast launch?
  yes → Pinecone (or Weaviate Cloud, Qdrant Cloud)
  no ↓
Already on Postgres, < ~10M vectors, need SQL joins?
  yes → pgvector
  no ↓
Prototype / local / embedded app?
  yes → Chroma or LanceDB
  no ↓
Self-host at large scale?
  → Qdrant, Milvus, Weaviate (evaluate ops capacity)

Common Failure Modes

Symptom	Likely cause	Fix
Irrelevant results	wrong embedding model or huge chunks	match model to domain; tune chunk size
Misses exact SKU/error codes	pure semantic weak on tokens	hybrid BM25 + vector
Slow queries	no ANN index, brute force	HNSW/IVF, reduce dimensions
Stale answers	index not updated	event-driven re-embed pipeline
Duplicates in top-K	overlapping chunks	dedupe, MMR diversification

Cost Sketch

Embedding once: 1M tokens × $0.02/1M (small model) ≈ $0.02
Storage: 1M vectors × 1536 dims × 4 bytes ≈ 6 GB raw
Pinecone: tiered by pods/storage; pgvector: your Postgres bill
Query: dominated by LLM if RAG—retrieval often cents per 1K queries

Cache embeddings—re-embed only when source text changes.

Minimal End-to-End Example (Conceptual)

# 1. Chunk and embed documents
chunks = split(document, size=512, overlap=64)
vectors = embed_model.encode(chunks)

# 2. Upsert to vector store (Chroma/pgvector/Pinecone—all similar)
store.upsert(ids=ids, embeddings=vectors, metadatas={"source": doc_id})

# 3. Query time
q_vec = embed_model.encode(user_question)
hits = store.query(q_vec, top_k=5, filter={"source": allowed_docs})

# 4. RAG generation
prompt = format_context(hits) + user_question
answer = llm.generate(prompt)

Summary

Piece	Role
Embedding model	text → meaning vector
Vector DB	store + fast similarity search
RAG	retrieve context for LLM
Semantic search	find by intent, not keywords
Recommendations	similar items via vector neighbors

Pinecone for managed speed, pgvector when Postgres is already your system of record, Chroma for dev and lean deployments. Pick the store for your scale and ops—but invest equally in chunking, hybrid search, and eval, or the best DB still returns the wrong paragraphs.

Next in the models track: revisit text, vision, and multimodal articles—or jump to case studies to see RAG in production (Shopify Sidekick).