The tools lie

So far the tools have always told the truth. Now they do not. Some calls fail, some come back cut short, and some return a clean-looking answer that is simply wrong. Your job is to build a harness that does not believe everything it reads.

Your goal: answer correctly even when roughly one tool call in seven goes wrong, by noticing and working around the bad results.

This problem is about reliability. New to this? Read Reliable tool use first. It explains the kinds of failure and how to check for them.

What you are looking at

This is the same job as problem 05: factual questions about a read-only copy of the pytest repository. Pytest is a popular Python test runner. You do not need to know anything about it. The same questions, the same grader, and the same budget apply.

The editor holds harness.py, the code around the model. It is the only file you edit. The model is gpt-4o-mini at temperature 0, the same for everyone.

The difference is the tools. list_dir, grep, and read_file are wrapped so that they misbehave.

How the tools misbehave

About 15% of calls are dirty. A dirty call fails in one of three ways:

• A visible error. The call comes back as an error, such as a timeout.
• A cut-off result. Part of the output is missing.
• A silent lie. The call returns another file's contents, or a plausible line that is not from the path you asked for. There is no ERROR: prefix, nothing to warn you. It looks like a normal answer.

The first two announce themselves. The third does not, and it is the one that costs you the score. A model reading a lie believes it, and so does a harness that only checks for the word ERROR.

The faults are repeatable. They are decided by the task and by the exact call, so the same call on the same task always goes wrong in the same way. A call with different arguments is a fresh roll.

What goes wrong

The starter trusts every result. It adds each one to messages and moves on. When a tool lies, the lie becomes part of the conversation, and the model builds its answer on it with full confidence.

Retrying on errors is ordinary engineering, and it is not where the score moves. The score moves when you stop trusting a single result. Ask whether it is about the path you asked for. Ask whether a second source agrees. A result that looks fine can still be wrong.

A prompt that says "be careful" will not help. The model cannot compare a result to the real repository. It can only read what is in the conversation.

Your task

1. Open Results and read the visible traces. Find a moment where a result looked fine and was not.
2. Decide what your harness could have checked to notice. For example: did the result match the request? Does it have the shape you expect? Does a second, independent read agree?
3. Add those checks to the loop, in code, before a result is added to messages.
4. Decide what to do when a check fails: ask again a different way, read from another route, or tell the model that a value could not be verified.
5. Keep an eye on your budget. Every extra check costs steps and tokens.

The code

from harness import llm, tools, Task

def run(task: Task) -> str:
    """Return the final answer as a string."""
    ...

You still import tools from the platform. You do not get a clean copy, so any cross-checking has to happen in your loop.

How the answer is checked

Return a short string and nothing else. The grader lowercases your string, strips most punctuation, and keeps the last non-empty line. An empty string fails.

Limits and scoring

• 15 steps. One llm.chat call is one step.
• 25,000 tokens for the whole task.
• 90 seconds per task.
• Going over any limit fails that task, even if the answer was right.
• Run tries your code on 3 visible tasks. You see each question, your answer, and the full trace.
• Submit tries it on 30 hidden tasks. You pass with 90% (27 of 30). The hidden questions stay sealed.