Remember what matters

Everything so far happened inside one run. Here the work is split across two sessions, and the second one starts from nothing. Only what you saved in the first session survives. Your job is to decide what to save, and how to find it again.

Your goal: in session 1, save the right facts under clear keys. In session 2, fetch only the entries you need, and answer.

This problem is about memory. New to this? Read Agent memory first. It explains the difference between context and memory, and what makes a good entry.

What you are looking at

The editor holds harness.py, the code around the model. It is the only file you edit. The model is gpt-4o-mini at temperature 0, the same for everyone.

Each task is an episode with two sessions. Your run(task) is called twice, once for each session, and task.session tells you which one you are in.

Session 1. You have the tools. A small workspace of text files is available: a roster, deploy notes, services, and on-call details, plus a few files that are out of date or unconfirmed. task.question is a short briefing, not a real question. Read what you will need, and write it into memory. What you return in this session is ignored.
Session 2. The files are gone. list_dir, grep, and read_file all return an error. A new run(task) starts with an empty conversation, and task.question is the real question. The only thing that survived is memory. Return the answer.

Only your session 2 answer is graded.

The memory store

memory is a simple key-value store, like a Python dictionary that survives between the two sessions.

memory.put(key, value) saves a value under a key.
memory.get(key) returns the value, or None if there is nothing under that key. Each get counts as one fetch.
memory.keys() returns the list of keys. Listing keys is not a fetch.

Keys and values are strings.

Decoys

Before the episode begins, the store has already been seeded with extra entries. Their keys look like the answer, and their values are wrong. For example, a question about the staging deploy key may find an entry under a key with almost the same name, holding a different value. You will see them in memory.keys().

The workspace has its own traps. Some files say "do not use", or "unconfirmed", or "heard in slack". They look relevant, and they are not the source of truth.

So there are two ways to go wrong: saving the wrong facts in session 1, and fetching the wrong entry in session 2.

What goes wrong

The starter writes everything. In session 1 it reads every file, including the outdated and unconfirmed ones, and puts it all into one key. In session 2 it fetches every key and gives all of it to the model. That looks busy, and it scores badly: the right value is buried among stale ones, and the decoys get a vote.

Dumping every file into one key, or fetching every key, also fails the task, even if the answer string happens to be right. Fetching 6 or more keys counts as a dump.

Your task

Open Results and read what the starter did, and what it got wrong.
Decide which files are trustworthy, and which are not.
In session 1, save small entries, one fact each, under names you can predict later. Skip the files that are not reliable.
In session 2, work out which key the question needs. Fetch that one, and check the value.
Return the value on its own. Press Run, and compare what you stored with what you fetched.

A good test is to imagine a stranger doing session 2. They have no files, no tools, and one question. What would you want to hand them?

The code

from harness import llm, tools, Task, memory

def run(task: Task) -> str:
    """task.session is 1 or 2. memory persists across the two sessions."""
    ...

How the answer is checked

Return a short string and nothing else. The grader lowercases your string, strips most punctuation, and keeps the last non-empty line. Some questions accept more than one phrasing of the same value, such as Mina Chen or Mina. An empty string fails.

Limits and scoring

15 steps. One llm.chat call is one step.
25,000 tokens for the whole task.
120 seconds per task.
Going over any limit fails that task, even if the answer was right.
Run tries your code on 3 visible episodes (session 1, then session 2). You see each question, your answer, and the trace.
Submit tries it on 20 hidden episodes. You pass with 85% on the session 2 answers. The hidden questions stay sealed.

Read the guide: Agent memory →See the problems