Start here · Harness Dojo

Before you start

You do not need to know anything about machine learning, and you do not need to have built with an AI API before. If you have written a Python function, looped over a list, and used a dictionary, you have everything you need. Nothing to install either: the editor runs in your browser.

Here is what you will do on this site, in one sentence. You are given a small program that asks an AI model a question and tries to get an answer. It is broken in one specific way. You find the break by reading a record of what happened, fix it, and see your score go up.

This page is long on purpose. Read it once, top to bottom, before problem 01. Later you can jump back using the list above. If a word is new, the glossary near the end explains it in a sentence.

The big idea

What a language model can and cannot do

A language model is the thing behind ChatGPT. Strip away the product and it does one job: you hand it a conversation, written as a list of messages, and it hands back one new message. That is all it does.

It cannot open a file. It cannot search the web, run code, or look at your project. It also remembers nothing between calls. Each time you call it, it sees only the messages you send in that call.

Picture a very well-read person locked in a room with no phone and no computer. The only way to communicate is by sliding notes under the door. They know only what is on the notes you slide in. If you send a new note and leave out the earlier ones, they have no idea the earlier ones existed.

The harness is the person outside the door

The harness is the code that stands outside that door. It decides what to write on each note. It goes and fetches things the person in the room asks for. It slides the answers back in, and it decides when the work is finished. The model is the same for everyone. The harness is where the engineering lives.

That is why this matters outside of puzzles. Coding assistants, support bots, and research agents are each a model plus a harness, and a lot of what separates a good one from a bad one is the harness. Same model, better harness, better results.

Tools: how the model gets hands

Since the model cannot open files itself, you give it a menu of tools. A tool is a named action, such as read_file or grep. The model cannot run a tool. It can only write a request that says "please run this tool with these arguments". Your harness runs the tool and sends the result back as the next note.

One question, start to finish

Here is a complete run, shortened. The question is real: it is one of the visible tasks in problem 01.

harness sendsInstructions + the question: "In .readthedocs.yaml, which operating system is the docs build set to use?"
model repliesA tool request, not an answer: read_file(path=".readthedocs.yaml")
tool runsThe harness runs it and gets the file text back. Near the bottom it says: build: os: ubuntu-24.04
harness sends againInstructions + question + the model's own tool request + the file text. All of it, again.
model repliesubuntu-24.04
harness returns"ubuntu-24.04" as the final answer.

Read the fourth row twice. The model was called two times, and the second call had to include everything from the first. The model did not remember asking for the file, because it remembers nothing. The harness had to repeat it. That one fact is behind most of the problems here.

An agent is a model plus a loop like this one. Ask, run whatever tools it requests, show it the results, ask again, and stop when it gives an answer. You will write that loop in a few lines.

What this platform is

HarnessDojo is practice for the code around a language model. The goal is that you can build a harness, see why a run went wrong, and change the loop until it holds up on questions you have not seen.

Everyone here uses the same model: gpt-4o-mini at temperature 0. Temperature 0 means the model picks its most likely reply each time, so the same input gives (almost always) the same output. That makes the score a measure of your code. A bigger model is not on offer, because then the score would measure the model.

The same skills are what applied AI, harness engineering, and evals interviews ask for: build a loop, debug a transcript, and explain why a run that got the right string was still a bad run. Practice here is practice for that work.

A tour of a problem page

Every problem is one page with two halves.

Left: Problem and Results. The Problem tab is the statement: what is broken and what to change. The Results tab already shows how the starter did on the visible tasks, with a full record of each one, before you spend a single Run.
Right: the editor. It holds harness.py, a starter harness that already works, except for the one thing the problem is about. You edit only this file. Your draft is saved in your browser. Reset starter puts the original back.

Those starter records are the real assignment. Read them, find the exact turn where things went wrong, and change the code so that mistake cannot happen on a question you have not seen.

Run and Submit

Run executes your current file on the visible tasks, usually three. You see each question, your answer, whether it passed, and the full record. Use it to experiment. No account is needed. It is capped at 20 an hour.
Submit executes your file on the hidden tasks, 20 to 52 depending on the problem. You see a score, not the questions. That percentage is the number that counts. Submit needs a GitHub sign-in.

Why hide questions? If you could see all of them, you could fix each one by hand and never improve the harness. The hidden tasks are different questions of the same kind, so only a real fix carries over.

Each account gets 5 Submits a day, and the count resets at midnight UTC. The header shows how many you have left. A shared network has its own cap, so the number can be lower than 5. The History tab keeps your earlier runs and the source you submitted.

The loop, line by line

Here is the whole idea in code. It is about fifteen lines and it is a complete, working harness. Think of it as hello world. Every problem on the site starts from a version of it.

In plain English: ask the model. If it wants tools, run them and show it the results, then ask again. If it just answers, return the answer. Give up after a fixed number of tries.

harness.py

1from harness import llm, tools, Task2 3def run(task: Task) -> str:4    # What the model sees. You decide what goes in here.5    messages = [6        {"role": "system", "content": "Answer with only the value."},7        {"role": "user", "content": task.question},8    ]9    for _ in range(task.budget.max_steps):10        resp = llm.chat(messages, tools=tools.schemas())   # one step11        if not resp.tool_calls:12            return (resp.content or "").strip()            # the answer13        messages.append(resp.message)14        for call in resp.tool_calls:15            result = tools.call(call.name, call.args)      # run the tool16            messages.append({"role": "tool", "tool_call_id": call.id, "content": result})17    return "unknown"

Do not worry if some of this looks unfamiliar. The next part goes through it one line at a time.

Each line

from harness import llm, tools, Task

This pulls in the three helpers the platform provides. llm is how you talk to the model. tools is the menu of actions the model may request, plus the code that runs them. Task describes the question you were given. The model, the grader, and the files stay hidden behind these three. You reach them only through these names.

def run(task: Task) -> str

The platform calls run once for each question, handing you a Task. You return a string, which is your answer. Only that returned string is graded. A print, or a fact you stored in a variable, does not count. The next sections cover how grading treats the string.

messages

This list is the conversation so far. Each item is a dictionary with a role and some content. The system item is your standing instruction. The user item is the question. This list is everything the model will know. If something is not in it, then as far as the model is concerned it does not exist. Deciding what goes in this list is most of the job.

for _ in range(task.budget.max_steps)

Repeat up to max_steps times. Each pass is one turn of the conversation. The cap is there so a confused model cannot loop forever. The _ is a Python habit for a counter you do not use. You do not count steps yourself: every llm.chat call counts as one, and going past the cap fails the task even if the answer would have been right.

resp = llm.chat(messages, tools=tools.schemas())

This is one call to the model, and it is called a step. You send the conversation and the tool menu. tools.schemas() describes each tool's name and arguments in a format the model understands. The model replies once, into resp. It can only request tools and arguments that appear on that menu, no matter what your instructions say.

if not resp.tool_calls

resp.tool_calls is the list of tools the model asked for. If it is empty, the model is not asking for anything. It wrote an answer, in resp.content. We strip extra whitespace and return it, which ends the task. The or "" is there because content is None when the model asked for tools, and we would rather get an empty string than a crash. This is the one normal way out of the loop.

messages.append(resp.message)

The model asked for tools. Before anything else, record what the model said by adding its message to the conversation. A tool result has to come after the request it answers. If you add the result alone, the next llm.chat is rejected: a tool message with no request in front of it.

for call in resp.tool_calls

One reply can contain several requests, such as reading two files at once. This inner loop deals with them one at a time. Each call has a name, an args dictionary, and an id. Every call needs its own tool message, so none can be skipped.

result = tools.call(call.name, call.args)

This line actually runs the tool. It is the only line that touches the repository. You pass the name and the arguments straight through, and you get a string back. If something goes wrong, such as a path that does not exist, you do not get a crash. You get text that starts with ERROR:, which the model can read and react to. Until you append it, result is just a variable, and the model has no idea the tool ran.

{"role": "tool", "tool_call_id": call.id, "content": result}

This puts the result into the conversation. role says it is a tool result, tool_call_id says which request it answers, and content is the text the tool returned. Even after this line, the model has not read it. It reads the list on the next pass of the outer for, when llm.chat runs again. A result you computed but did not append is, from the model's side, a tool that never ran.

return "unknown"

You only get here if the for finishes without the model ever answering. Every allowed step was spent on tool calls. unknown will not match any real answer, so the task fails. If you see it in your results, it usually means the model kept making the same request, because nothing new was put in front of it between calls.

The pieces you are given

Four small objects show up in the loop. Each is a handful of fields. This section is for reference, so skim it now and come back when you need it.

task

One question, plus the budget for answering it.

task

1task.question            # str. The question for this task.2task.budget.max_steps    # int. How many llm.chat calls you may make.3task.budget.max_tokens   # int. How many tokens the whole task may use.4task.session             # int. 1 or 2. Only the memory problem changes it.

question is the text in the user message. budget.max_steps is how many llm.chat calls this task allows, and budget.max_tokens is how many tokens the whole task may use. A token is a small chunk of text, roughly three quarters of a word, and models count length in tokens. session is 1 or 2. Only the memory problem uses it. On every other problem it stays 1.

tools

tools.schemas() is the menu: a list of dictionaries, one per tool. You hand that list to llm.chat. Each entry is a description of one tool, like this:

tools.schemas()

1{2    "type": "function",3    "function": {4        "name": "read_file",5        "description": "Read a text file. Optionally slice by 1-based line numbers.",6        "parameters": {7            "type": "object",8            "properties": {9                "path": {"type": "string"},10                "start": {"type": "integer"},11                "end": {"type": "integer"},12            },13            "required": ["path"],14        },15    },16}

The properties are the arguments the model is allowed to send. If an argument is not listed, the model cannot use it. Keep that in mind, because one of the early problems depends on it.

tools.call(name, args) runs one tool and returns a string. The usual tools are list_dir (list a folder), grep (search text), and read_file (read a file, optionally a range of lines). A miss comes back as text starting with ERROR:, not as an exception.

resp

One reply from llm.chat.

resp

1resp.content      # str or None. The answer, when the model is not calling tools.2resp.tool_calls   # list of calls. Empty when content is the answer.3resp.message      # dict. The assistant message. Append this first.4resp.usage        # prompt_tokens, completion_tokens, total_tokens.

content is the answer text, or None when the model asked for tools instead. tool_calls is empty when it wrote an answer. message is the assistant dictionary you append to the conversation. usage counts tokens: prompt_tokens for what you sent, completion_tokens for what came back, and total_tokens for both.

call

Each item in tool_calls is one request:

tool call

1call.id      # str. Goes on the tool message as tool_call_id.2call.name    # str. "list_dir", "grep", or "read_file".3call.args    # dict. Already parsed. {"path": "pyproject.toml"}

args is already a dictionary, so you do not parse anything. On the evaluator problem you read a saved transcript instead of a live reply, and there the arguments are still a JSON string. That problem's statement points out the difference.

Two problems use a different shape. Build the evaluator asks you to write evaluate(trajectory) instead of run. Remember what matters adds a memory store. The statement on each one says so at the top.

What the problems change

Two Python statements do most of the work in the loop, and nearly every problem comes down to one of them.

for gives the model another turn. A model only gets a second look at your tool results if you call llm.chat again. A harness that calls it once and then returns has no loop, so the model never gets to use what the tool found.
if decides what this turn was. No tool calls means return the answer. Tool calls means run them and go around again. A second if inside that branch is where the harness can make real decisions. It can look at result before appending it and drop lines from the wrong folder. It can retry when a result starts with ERROR:.

You can also change what the model is allowed to ask for. If read_file lists only path, the model cannot request a line range. Copy the schema, add start and end, and pass your version to llm.chat in place of tools.schemas().

The first three problems are this exact loop with one piece missing. Read the starter next to the hello world above, then read one visible record, and you can see which line never did its job. Later problems point the same two statements at harder situations: noisy search results, a tool that lies, a menu with too many tools, or a fact that has to survive into a second session.

Reading a trace

A trace is everything that happened on one task, in order. Think of it as a replay. The Trace tab shows it as cards: the system prompt, the question, each tool the model called and what came back, and the model's final answer. Use the full-screen button on a card, or on the whole trace, to read long results.

The Result tab is the summary. It shows what your harness returned, what the expected answer was, and why the task failed.

Most fixes come from reading one failing trace closely. Do not start with the score. Start by asking:

What did the model ask for on each turn? Is it the same request repeated?
What did the tool send back? Was it what the model needed, or was it empty, huge, or misleading?
On the next turn, did the model see that result? If the model behaves as if it never got it, the result probably never made it into the conversation.
How did it end? Did the model write an answer, or did the steps run out and the harness return unknown?

Somewhere in that sequence there is a first turn where the model saw the wrong thing, or never saw the right thing. That turn is what you are fixing.

How you're scored

Each task passes or fails. Your score is the percentage of hidden tasks that pass. Each problem sets a bar, 85% or 90%. Clear the bar and the problem is solved.
Each task has a budget. Steps, tokens, and seconds. Going over any one fails the task, even if the answer was right.
Return the value, not a sentence about it. Before comparing, the grader lowercases your string, removes most punctuation, and keeps only the last non-empty line.

That last rule catches many first attempts. Suppose the expected answer is >=3.10. If your harness returns >=3.10, it passes. If it returns The requires-python value is ">=3.10"., it fails, even though the right value is inside the sentence. The starter's system prompt already asks the model to reply with only the answer, and it is worth keeping.

Your first session

Here is the order I would use on problem 01, and on every problem after it. The visible records are there so that Submit is a check and not a search.

Read the statement until you can say, in your own words, which line of the loop is wrong, missing, or too coarse.
Open Results and pick one failing starter task. Ignore the score. Read its trace and find the turn where the model saw the wrong thing, or never saw the right thing.
Make a guess about the cause, and change one thing. One if, one for, or one field on a tool. Small changes are easy to understand.
Press Run. Put the new trace next to the old one. The turn you meant to fix should now look different. If it does not, your guess was wrong, and that is useful to know.
When the visible tasks pass for the reason you intended, Submit. A prompt tuned to the three questions you can see will miss the hidden ones, because they ask something else.

If you are stuck: read the comments inside the starter file. They usually point at the broken part. Open the trace and count how many times the model made the same request. If your edit made things worse, press Reset starter and try a smaller change. If Run shows a runtime error, open the trace to see the stack.

Glossary

model: The language model. Here it is always gpt-4o-mini at temperature 0. It reads messages and writes one message back.
harness: The code around the model: the loop, the tools, what goes into the conversation, when to stop. It is the file you write.
agent: A model running inside a loop that lets it use tools and see the results.
message: One item in the conversation: a dictionary with a role (system, user, assistant, or tool) and content.
step: One call to llm.chat. Each task has a step cap.
tool: A named action the model can request, such as read_file. The harness runs it, not the model.
schema: The description of a tool's name and arguments, shown to the model. It decides what the model is able to ask for.
token: A small piece of text, around three quarters of a word. Models and budgets count in tokens.
context: Everything the model can see on a single call. In this loop, that is the messages list.
task: One question you must answer, with its budget.
trace: The full record of one task: every message, tool call, result, and the final answer.
visible / hidden: Visible tasks are the ones Run uses and shows you. Hidden tasks are the ones Submit uses and keeps sealed.
budget: The limits on steps, tokens, and time for one task.
eval: A test that scores how well a harness or model does. Submit is an eval, and problem 06 asks you to write one.

The problems

They get harder as you go. Each one builds on the loop above.

01The tool result goes missingFoundations
The model asks for a tool, but the result never reaches it. Put the result back into the conversation.
02The second call never happensFoundations
One tool call is not a finished task. Let the model take another turn with the result in front of it.
03The read has no windowFoundations
Reading a whole file wastes the budget. Teach the model to ask for just a few lines.
04Old results never leaveLoop & Context
Every record the model reads is sent again on every step. Shrink the ones it has finished with.
05Keep the look-alikes outLoop & Context
Searches return look-alike lines from docs and tests. Keep the misleading ones out of the conversation.
06Build the evaluatorEvals
Decide whether an agent run was actually good, not just whether the final answer matched.
07The tools lieRead the spec
Tools sometimes fail, cut output short, or return the wrong file without saying so.
08Kill the toolsetRead the spec
Twelve messy tools is too many. Give the model at most five good ones.
09Remember what mattersRead the spec
Save the right facts in one session and find them again in the next.

Start with 01. The later problems build on it.