Reliable tool use

Think of tool failures along two lines. Does the failure announce itself? And is the data partly useful or completely wrong?

• A loud error. The tool says it failed: ERROR: file not found, a timeout message, a rate-limit notice. You know something is wrong.
• Truncation. The result is real but cut short. It may stop in the middle of a list, or end with "...". The part you needed may be in the missing portion.
• Empty or partial results. The tool returns nothing, or less than you asked for, without any note about it.
• A silent lie. The tool returns a well-formed, plausible result that is wrong: data for a different item, a stale cached copy, a neighboring row.

The first is the easy one. The last is the dangerous one, because nothing in the result tells you to be suspicious. A model reading it will believe it, and so will a harness that only checks for the word ERROR.

It is tempting to add a line to the instructions: "Be careful. Double-check tool results." The model will agree and then do what it always does, which is to read the text in front of it and continue.

A model cannot compare the result against the world, because it cannot see the world. It can only see the list. If a tool says the capital of Peru is Quito, nothing in the list contradicts it. Detecting that kind of problem means giving the model a second, independent piece of evidence, or checking something outside the model entirely. That is the harness's work.

For loud failures, the goal is to get what you need without wasting the budget.

• Retry what is safe to retry. A timeout on a read is usually harmless to repeat. Use a small, fixed number of attempts, such as two or three, and then stop. Endless retries just burn steps.
• Change the request when truncation is the problem. If the output was cut, ask for a smaller piece: a narrower range, a more specific search, a smaller page.
• Tell the model what happened. If a tool failed and you could not recover, put an honest note in the list: "read failed twice, so this value is not verified". Do not leave a silent gap, and never fill it with a guess.
• Know when a retry is unsafe. Reading is repeatable. Actions such as sending an email or charging a card are not. Retrying those can do the thing twice.

A silent lie needs a check. The idea is simple: before you trust a result, test it against something you already know or something independent.

Check that it is what you asked for

Compare the request to the response. If you asked for the weather in Oslo, does the result mention Oslo? If you asked for one file and the content starts with a different file's header, you have a mismatch. These checks are cheap and catch a surprising amount.

1def weather_for(city: str, result: str) -> bool:2    """A forecast is only usable if it is actually about the city we asked for."""3    return city.lower() in result.lower()

Check that it has the right shape

Most results have a format. A price has digits. A date parses. A forecast has a temperature. A result that is the wrong shape, such as an empty string, a stack trace, or text that is unrelated to the question, should not go into the list as though it were data.

Check it against a second source

For anything important, get the same fact a second way. Use a different tool, a different query, or a different part of the same source. If two independent routes agree, you can be much more confident. If they disagree, you have learned something valuable before the model has committed to anything.

1def convert(amount, rate_from_tool, rate_from_table):2    """Use the tool's rate only if it agrees with an independent reference."""3    if abs(rate_from_tool - rate_from_table) / rate_from_table > 0.05:4        return None   # disagree: do not trust either yet, look closer5    return amount * rate_from_tool

Every check you add uses a step or some tokens, and tasks have budgets for both. You cannot verify everything, so decide where it pays.

• Verify facts that the final answer depends on directly.
• Be more careful when a mistake would be expensive.
• Skip checks on information that is only background.
• Fold checks into code where you can. A comparison in Python is free. A second model call is not.

The best verification is often written once as ordinary code in the harness, running on every result, and costing the model nothing.

Be careful to separate "the tool failed" from "the tool worked and the answer is no". An empty search result could mean nothing matched, or it could mean the search broke. The two call for different responses. If you collapse them into one case, a harness will either give up too early or keep going after it should have stopped.

A tidy tool returns different messages for them: "0 matches for 'invoice' in billing/" versus "ERROR: search timed out". If a tool you are given does not, make the difference visible in your own wrapper.

When your harness retries, rejects, or re-checks, write it down in the trace. A short note such as "result rejected: path did not match request" is what lets you debug the run later, and it gives the model an accurate picture of what happened. Silent corrections make traces hard to read and bugs hard to find.

1. Did the tool report an error? If so, is a retry safe and does it have a limit?
2. Was the output cut off, and do I have everything I need?
3. Is it about what I actually asked for?
4. Does it have the shape I expect?
5. Does it matter enough to confirm with an independent source?
6. What will I tell the model if I cannot trust it?

Problem 07 puts you in front of tools that sometimes fail loudly, sometimes cut things short, and sometimes return another item's data with no warning. A prompt that asks the model to be careful will not change what the tools return. Read the visible traces for the moments when a result looked fine and was not, then decide what the harness could have checked to notice.