Case Study: Shopify's RAG Path to AI Support

“RAG support bot” is the most common AI use case—and among the most failed. Shopify’s Sidekick is a rare public example with engineering path and metrics.

Background

Millions of merchants; 10,000+ Help Center articles; support drowning in repeats:

• “How do I set shipping rates?”
• “Why did my payment fail?”
• “How do I add a custom domain?”

~80% answered in docs—merchants still can’t find or parse them.

Phase 1: Pure RAG MVP (2023)

merchant question → retrieve Help Center → LLM answer

V1 problems:

Issue	Data
Bad retrieval	wrong doc ~40%
Vague answers	”said nothing useful”
Hallucination	~15% not in docs
No actions	answer only

Satisfaction ~60%—not shippable quality.

Phase 2: Hybrid Search + Structured Answers (2024)

Retrieval

v1: vector only
  ↓
hybrid = vectors + BM25 + metadata filters
  ↓
reranker: top 20 → top 3

Accuracy 60% → 85%.

Structured Output

Template instead of free prose:

## Answer
[direct answer from docs]

## Steps
1. Settings → Shipping
2. Manage rates
3. …

## Sources
- [Shipping setup guide](link)

## Need more help?
[Talk to support]

Hallucination 15% → ~3%—format constrains the model.

Phase 3: RAG + Agent (2025, Sidekick)

Sidekick executes, not only explains:

Merchant: "Set US standard shipping to $5.99"

Sidekick:
  1. [RAG] shipping setup docs
  2. [Agent] Shopify Admin API → create rate
  3. "Done—US standard shipping $5.99. [View details]"

From “how to” → “done for you”.

Architecture

┌─────────────────────────────────────────┐
│              Sidekick Agent              │
│  ┌─────────┐  ┌──────────┐  ┌────────┐ │
│  │ RAG      │  │ Admin    │  │ Store  │ │
│  │ (docs)   │  │ API      │  │ Context│ │
│  └─────────┘  └──────────┘  └────────┘ │
│  merchant profile + store state + history │
└─────────────────────────────────────────┘

Store Context—apps installed, config, recent orders—is live business data, not RAG.

Published Metrics

Metric	MVP (v1)	Sidekick
Auto-resolve rate	30%	82%
Satisfaction	60%	91%
Avg response	8s	2s
Hallucination	15%	< 2%
Human ticket volume	baseline	−40%

Engineering Lessons

1. Don’t ship v1 at 60% satisfaction—brand risk
2. Hybrid > vector-only—BM25 for exact terms
3. Structured output > free text—format = quality guardrail
4. RAG + agent + business context—full stack
5. Always offer human escalation

Reusable Playbook

Any “docs Q&A + operations” product:

Phase 1: RAG Q&A (prove retrieval > 80%)
Phase 2: structure + rerank (satisfaction > 85%)
Phase 3: agent tools (auto-resolve > 70%)
Phase 4: live business context (personalization)

References

• Shopify Sidekick
• Shopify Engineering — AI
• Shopify Editions 2024/2025 keynotes

Summary

Shopify shows RAG support is not plug-and-play API. Retrieval, output shape, agents, and store context each need engineering—and 82% auto-resolve is achievable.