Kill the toolset

The model is fixed. This time you design the menu of tools it gets to use. The platform hands you twelve overlapping, badly named tools, and the model is only allowed to see five. Your job is to choose, name, and describe them well.

Your goal: show the model at most five tools that cover the whole job, and make each one hard to misuse.

This problem is about tool design. New to this? Read Tools and schemas and then Designing a tool set.

What you are looking at

The job is the same as problem 05: factual questions about a read-only copy of the pytest repository. Pytest is a popular Python test runner. You do not need to know anything about it. The questions, the grader, and the budget are the same.

The editor holds harness.py, the code around the model. It is the only file you edit. The model is gpt-4o-mini at temperature 0, the same for everyone.

What changes is the menu. In this problem tools.schemas() returns all twelve tools:

search_code, find_in_repo, scan, and lookup all search, in slightly different ways.
search_docs and search_tests search only one folder each.
read, read_lines, open_file, and cat all read a file. Some take a line range and some do not.
list_files and ls both list a folder.

Several of these do almost the same thing. Several are slightly wrong for the job. If you show the model all twelve, it will often pick the wrong one, and the long menu also costs tokens on every call.

The rule

You may show the model at most five tools. If you pass more than five schemas to llm.chat, the call raises an error and that task fails.

You write the schemas. Behind them, you can call any of the real tools with tools.call(name, args). The model never needs to know that a mess exists underneath. A tool you show can be a wrapper that picks the right underlying tool, fills in sensible defaults, and returns a clean, consistent result.

What goes wrong

The starter hands the whole menu to the model. That is already a fail: with twelve tools the model has twelve ways to choose, several of them wrong, and with more than five schemas the call is rejected anyway.

A better prompt will not fix this. A great prompt on a bloated menu loses to an average prompt on a tight one. The score moves because you decided what the model is allowed to call.

Your task

Open Results and read the visible traces. Which tools did the model reach for? Which ones overlapped?
Work out the small set of jobs the questions need. For example: list a folder, search, read a range of lines.
Write one clear schema for each job, with a plain name, a description that says when to use it, and only the arguments it needs.
Write the code that turns each call into the right underlying tool.
Pass only your schemas to llm.chat, and press Run. Compare the new traces to the old ones.

The code

from harness import llm, tools, Task

def run(task: Task) -> str:
    """Return the final answer as a string."""
    ...

Build your own list of at most five schemas, and pass it to llm.chat instead of tools.schemas(). When the model calls a tool, run it with tools.call(name, args), or with your own wrapper, and add the result to messages as usual.

How the answer is checked

Return a short string and nothing else. The grader lowercases your string, strips most punctuation, and keeps the last non-empty line. An empty string fails.

Limits and scoring

At most 5 tools shown to the model.
15 steps. One llm.chat call is one step.
25,000 tokens for the whole task.
90 seconds per task.
Going over any limit fails that task, even if the answer was right.
Run tries your code on 3 visible tasks. You see each question, your answer, and the full trace.
Submit tries it on 30 hidden tasks. You pass with 85%. The hidden questions stay sealed.

Read the guide: Designing a tool set →See the problems