Production-Ready Evals

Your AI agent is live. Thread length is low. Retention is dropping. No error fired, no exception thrown. You just have no signal.

That's the problem nobody talks about. Not a crash — a silence.

Most engineers reach for a third-party eval tool at this point. Beautiful dashboard, generic scores flowing in. Relevance: 8/10. Coherence: 9/10. Looks like progress. But those metrics know nothing about your domain or your users. You went from no visibility to fake visibility.

Two Types of Failures

Hard failures — exceptions, tool call failures, malformed JSON structured outputs. Your code already sees these.

Write fast boolean assertions on format, schema, and tool responses. Run them on every change. No LLM needed, zero cost per run.

Soft failures — the agent responds. It just responds wrong. Off-tone, misses intent, gives a plausible-sounding bad answer. Code can't catch these.

The Eval You Already Have

Here's what most people miss: you already have an eval for soft failures. You just never ran it.

Go into your chat logs. Classify the frustrated conversations — short threads, repeated rephrasing, abrupt drop-offs. Those users were already telling you what's broken. Cluster the failure patterns that surface.

Building Ground Truth

Pull the top 200 most frequent queries from your system. Hand 50–100 to a domain expert — not you the engineer, someone who knows what a correct answer looks like for this domain.

• Binary labels
• One-line reasoning each

That dataset is your golden standard.

LLM-as-Judge → Fix → Lock It

Build an LLM-as-judge aligned to those labels. Run it on the same pairs. Measure agreement score. Iterate the judge prompt until it thinks like the domain expert does.

Then run it against your failure clusters. Fix the agent. The numbers move. You have proof, not a feeling. Lock the fix with a unit test so it never regresses silently.

The Full Evolution

nothing
  → catch what breaks        (unit tests on hard failures)
  → measure what degrades    (LLM-as-judge on soft failures)
  → fix with proof           (ground truth + eval loop)

How did your eval setup evolve in the last AI agent you worked with?