Evaluating agents

An eval is a test for an AI system. You give the system a set of tasks, record what it does, and score the results. The Submit button on this site is an eval of your harness. In problem 06 you write an eval yourself.

Evals matter because AI systems are hard to judge by feel. A change that fixes the one example you were looking at can quietly break ten others. A score over many tasks tells you whether a change really helped.

Imagine a refund bot. A customer asks whether order 1187 can be refunded. The correct answer is "yes". Consider three runs that all end with "yes".

• Run A looks up order 1187, reads the refund policy, checks the delivery date against the window, and answers "yes".
• Run B never calls a tool. The model guesses "yes", because most refund questions have that answer.
• Run C calls eleven tools, mostly the same lookup repeated, then answers "yes".

Scored on the answer alone, all three pass. Only Run A did the job in a way you would trust. B got lucky and will fail when the true answer is "no". C works, but it is slow and expensive, and something in its loop is broken.

This is why agent evals often look at the trajectory: the full record of a run, including every message, tool call, and result. The answer is one piece of evidence. The path is another.

There are two families of check, and good graders usually combine them.

• Outcome checks ask whether the result is right. Did the answer match? Was the file created? Did the tests pass?
• Process checks ask whether the run earned that result. Did it consult the right source? Did it avoid forbidden actions? Did it stay within a reasonable budget?

Outcome checks are simple and hard to argue with. Process checks catch the lucky and wasteful runs. Using only outcomes rewards luck. Using only process rewards ritual, where an agent can follow the right steps and still get the answer wrong.

Before you write any code, name the ways a run can be bad. For agents the same few patterns appear again and again.

• The lucky guess. It searched, found nothing relevant, and guessed correctly anyway.
• Answered from memory. It made no tool calls, and relied on what the model already believed.
• Thrash. It made many calls that did not move it forward before getting to the answer.
• The wrong source. It relied on something unreliable, such as an example or an old draft, and happened to be right.
• A forbidden or unsafe action. It did something it should not have, such as refunding without checking.

Each of these leaves evidence in the trajectory. Your job is to say what that evidence looks like.

A good grader starts with one question: what must a good run show? Then it turns that into things you can check in the transcript.

Take the refund bot. A good run must have looked up the order, and must have consulted the policy. In the transcript, that means:

• an assistant message that requested get_order with the right order id,
• a tool message after it that returned order data,
• an assistant message that requested the policy,
• and a tool message after that, whose content contains the rule the answer depends on.

Notice that "a tool was called" is not on the list. A call by itself proves little. What proves something is that the right call was made, with the right arguments, and that what came back supports the answer.

A tool interaction has two halves, and a grader should look at both.

• The request is on an assistant message. It says which tool was called, and with which arguments.
• The result is on a later message with the tool role. It says what actually came back.

Here is a small example of walking a transcript to collect both.

1import json2 3def tool_calls(transcript):4    """Yield (name, args, result) for each tool call in a run."""5    results = {m["tool_call_id"]: m["content"] for m in transcript if m.get("role") == "tool"}6    for message in transcript:7        for call in message.get("tool_calls") or []:8            name = call["function"]["name"]9            args = json.loads(call["function"]["arguments"])10            yield name, args, results.get(call["id"], "")

With that helper, checks become short and readable: was get_order called for this order id, and did its result mention the order? A path or id in the arguments says what the agent intended. The result says whether it got it.

The record also holds counts: how many steps, how many tokens. These help, but be careful with them.

A count tells you how much work happened, not whether the work was good. A run with twelve steps might be thrashing, or it might be a hard question that needed twelve steps. A run with two steps might be efficient, or it might have skipped the evidence.

Use counts as one signal among others, or as a tie-breaker, and not as the whole rule. If the only thing separating good runs from bad ones in your data is a step count, you probably have a fingerprint of your examples and not a real test.

When your grader calls a run good or bad, it can be wrong in two ways.

• A false positive is a bad run that you called good. You let a lucky run through.
• A false negative is a good run that you called bad. You rejected honest work.

You will almost never get both to zero, and you should check both. A grader that is stricter catches more bad runs and also rejects more good ones.

Before you trust a score, compare it to the dumb baselines. If 70 percent of the runs are bad, then a grader that always says "bad" scores 70 percent without looking at anything. Your grader needs to beat that clearly to have learned something. Always ask what a trivial rule would score.

The most common mistake is to look at the labeled examples you were given and write rules that fit exactly those. It works perfectly on the examples and falls apart on new runs.

A few ways to guard against it:

• Write the standard in words first. If you cannot explain why a run is bad, the rule is probably an accident of the data.
• Check your rule against a run that is good but looks different: more steps than usual, or a different path to the answer.
• Check it against a run that is bad but looks plausible: it visited a relevant-sounding file and got nothing from it.
• Prefer rules that mention evidence (what was searched, what came back) over rules that mention superficial features (how many calls).

Code is not the only grader. You can also use another model as a judge, giving it the transcript and a rubric. That handles questions that are hard to turn into code, such as "was the explanation clear?". It also adds cost and some noise, and it can have its own biases. A common pattern is to use plain code for everything you can check exactly, and a model only for the rest.

Whichever you use, keep a short reason with each verdict. When a grader says "bad: no policy lookup", you can look at the run and see whether it was right. A grader that only returns a yes or no is very hard to debug.

Problem 06 asks you to write a grader, not an agent. You get recorded runs where every answer is correct, and some of those runs you would not want in production. Start by reading the labeled examples and writing, in plain words, what separates the good ones from the bad. Then turn each sentence into a check against the transcript, and test whether it holds for runs that look a little different.