Back to AI/ML Overview
🎯 Knowledge Site · Evals for Agentic Software · From One Test to Release Gates

Eval EngineeringA test checks an answer. An eval measures a behavior.

An agent can give a different answer each time, take a different path to the same answer, and be right for the wrong reason. This site starts from one test with one expected result and climbs, one idea at a time, to rubrics, golden datasets, trace grading, judges, repeated trials and release gates. At each step it asks one question: what does this number let me claim?

🎯The problem

A unit test runs once and gives the same result every time. An agent does not. The same input can give a different answer on the next run. Two runs can give the same answer, and only one of them checked the rules first. A team that tests an agent the way it tests other code gets a green result that says very little.

🔑Three sentences the site holds to
A score is an observed result. Confidence is a claim about uncertainty. Reliability needs repeated evidence. Every page keeps these three apart, because most wrong release decisions come from reading one of them as another.

🗺️What is inside

From plain words to rubrics

Task, case, trial, grader, rubric, score, threshold: each word gets one plain sentence and one small example before it is used. Then code, model and human graders, weighted scores, and the failures that no good prose may outweigh.

The answer, the path, and what was left behind

Output, trajectory and end state are three different things to grade. Traces and spans, a model as a judge and how to calibrate it, and five numbers that look alike and are not the same.

One run proves little

Repeated trials, at least one of k against all of k, intervals and sample size, a paired comparison with a baseline. Then the runner, the suites that may block a release, and on production traffic.

One example, with code that runs

A synthetic underwriting agent with ten cases, followed on every page. A small Python harness that runs with no model, no network and no key. Eleven interactive labs, and seven tools read on their own documentation.

🖱️How to read it

  • New to : read in order. Each page starts in plain English and hands on to the next.
  • Experienced: jump to traces, judges, reliability or CI gates. The rigorous part sits in folded blocks.
  • Every lab has a table with the same information next to it. Every diagram is clickable.
  • The example is synthetic and its agent is a scripted stand-in. The results show how works, not how good any model is.