Eval and Observability: Knowing When Agents Get Better

Traditional software has unit tests. AI agents need eval—yet most teams stop at “try a few prompts manually.” Here is how to build real quality systems.

Why Agents Are Hard to Test

Traditional:  input → f() → output        ✅ deterministic asserts
LLM call:     input → LLM → output        ⚠️ same input, different output
Agent:        goal → loop(tools×N) → result ❌ variable path and step count

Challenges:

1. Non-unique outputs—many valid bug fixes
2. Variable paths—3 steps vs 15 steps
3. Side effects—which file edits were correct?
4. Regression—prompt/model/skill change breaks old cases

Three Eval Levels

L1: Output Eval

Input → check output properties:

def eval_summarize():
    result = agent.run("Summarize three key points from this article")
    assert len(result.bullet_points) == 3
    assert result.language == "en"
    assert no_hallucination(result, source_doc)

Good for: RAG Q&A, generation, classification.

L2: Process Eval (Agent-Specific)

Inspect trace, not only final text:

def eval_code_fix():
    trace = agent.run("Fix 404 on login page")

    assert trace.used_tool("read_file")
    assert trace.used_tool("grep")
    assert "auth.ts" in trace.files_modified
    assert trace.total_steps <= 15
    assert trace.tests_passed

Good for: coding agents, workflows.

L3: End-to-End Eval

Near-production conditions:

Real repo + real bug → agent fix → CI green?
Real support question + real KB → answer → human score ≥ 4/5?

Expensive; closest to truth.

Building a Test Set

Start with 20 Cases

easy/         (5)  single tool, 1–3 steps
medium/       (10) multi-step, 5–10 steps
hard/         (3)  cross-system
adversarial/  (2)  misleading inputs

Each case:

name: fix-login-404
input: "Users report /login returns 404—please fix"
expected:
  files_modified: ["src/pages/login.astro"]
  tests_pass: true
  max_steps: 15
grading:
  type: process + output
  rubric: correct file changed and tests pass

Sources

1. Production failures (highest value)
2. Bad user feedback → reproduce
3. Synthetic variants (LLM expands set)
4. Regression—every bug fix adds a case

Online Monitoring

Offline eval is a snapshot; production needs:

- task success rate
- average steps (spikes = regression)
- tool call failure rate
- token spend
- human intervention rate (user edits/cancels)
- latency P50/P99
- explicit negative feedback rate

Example Alerts

Task success < 80% (1h window)     → page
Avg steps > 2× baseline (24h)      → Slack
Token spend > 3× daily avg (1h)    → cost alert
Same user 3 negative ratings        → human review

Tooling

Tool	Fit	Notes
LangSmith	LangChain users	traces, datasets, A/B
Braintrust	general	eval + regression + prod
W&B	ML teams	experiment tracking
DIY	small teams	Postgres + dashboard

Start on LangSmith or Braintrust free tier; build custom when scale demands.

Cursor-Style Eval (Inferred)

• Tab completion: implicit accept/reject signals
• Agent: undo rate, completion rate, step distribution
• Model A/B across cohorts
• Nightly internal agent task suites

Eval-Driven Development

Change prompt / skill / MCP / model
    ↓
Run offline suite (20–100 cases)
    ↓
Pass rate ≥ baseline? — no → rollback, analyze failures
    ↓ yes
5% canary
    ↓
Online metrics OK? — no → rollback
    ↓ yes
Full rollout + monitor

Iterating agents without eval = driving blind.

Summary

Stage	Work	Minimum effort
Start	20 manual cases, run on every change	~1 day
Grow	automate + LangSmith/Braintrust	~1 week
Mature	online monitors + regression + A/B	ongoing

Models change; harnesses evolve; eval is ground truth.