Nobody Told the Agent What Done Means

Domain expertise, not the job title

On 16 June 2026, Hitzig and colleagues published Agentic coding and persistent returns to expertise. In code-producing sessions, software-related occupations reach verified success about 34% of the time; other professions, about 29%. A five-point gap. Every one of the ten largest occupations lands within seven points of software engineers. Management occupations are highest, slightly above software engineering — with a caveat Anthropic flags: verification rests partly on explicit confirmation in the transcript, and managers may be more likely to say when they got what they asked for.

Verified success means the session was judged successful and showed at least one hard signal: git commits or pull requests matching the work, test suites passing, or explicit user affirmation. They cannot measure whether the code is used in production. The classifiers read transcripts.

That is nearly the same rate. A non-technical person who knows what correct looks like — a clear definition of done — can ship work on par with engineers.

Expertise, in their classifier, is task-specific, not a job title. The signals are how precisely the user frames directions, what they ask Claude to verify, and whether the user corrects Claude or Claude corrects the user. An accountant who has never used Python, but tells Claude exactly which reconciliation rules a script must enforce and catches the month-end edge case, is expert at that task. A senior engineer asking a first Rust question is a novice at Rust.

Novice verified success sits around 15%. Intermediate-or-up sits at 28–33%. Most of the gain is novice to intermediate. You need to know the problem well enough to say when the work is finished, not mastery.

In their words: "Success is determined by how well a person understands the problem they are trying to solve, not whether they're trained in coding."

People keep the planning decision

Planning, in the study, is what to do, which approach to take, and what counts as done. Execution is which files to change, what code to write, which commands to run. People make about 70% of the planning decisions and only about 20% of the execution. The agent will write the code. The person still has to decide what finished means.

That is a leverage problem first. The passing condition is the one place AI actually moves money. If you cannot say, in ordinary business terms, when the work is finished, you are not installing an agent. You are buying motion.

It is a memory problem second. What "done" means is company knowledge — the same asset I wrote about in what your organization remembers. If it lives in one person's head or a private chat, the agent invents a grader. That is not malice. It is how the system was trained.

Sessions with "no clear goal" were about 7.7% of Anthropic's sample. They were excluded from the success analysis. Leave the pass condition unwritten and you are volunteering for that bucket.

The harness already negotiated "done"

Anthropic had already shown the operator version of this. In their 24 March 2026 harness for long-running apps, a generator and an evaluator negotiate a sprint contract: they agree on what "done" looked like for that chunk of work before any code was written. The evaluator then clicks the live app with Playwright. Self-evaluation is a trap. An agent grading its own work will praise it.

Nate B. Jones put the same fact in operator language on 30 August. Inside "agent school," the finish line is everywhere. Then the agent graduates into a company that never built the exam. Agents chase a passing score. Most companies never wrote one in the measures they already use.

Hugging Face and Runable are later pictures of invented graders, not the frame. Evaluation agents joined an attack nobody assigned because they were trying to pass. TechCrunch asked Runable for a coffee-subscription site and 100 visitors; it built the site, prepared ads, and stopped at an advertising account that had never been connected. Grade emails sent, tickets closed, or tests passing, and the agent will move that score. The dashboard improves. The business does not.

This is uncertified finish. In when AI adoption hides what your team knows the failure is people keeping the useful bits where the system cannot reach them. In chat logs are not what we know the failure is uncertified chat becoming memory. Here nobody is hiding, and the transcript is not the point. The agent treats a plan, a status update, or a passing-looking artifact as done because nothing in the session is allowed to say no.

A definition of done the agent cannot rewrite

Before the agent writes code, sends mail, or produces a folder of files, write a card the agent is not allowed to edit. Anthropic's expertise signals are that card: precise framing, named verification, the user as the one who corrects.

What changed in the world. Not "research complete" or "emails sent." An observable outcome a second person could notice without sitting in the session.

The ordinary measure. One number the business already uses. If the agent's dashboard improved and this number did not, believe the ordinary measure.

Inspectable evidence. A test command and its exit code. A screenshot of the live page. A named reviewer who opened the file. Not the agent's summary of its own work. The evaluator, not the generator.

Who signs. A person, not "the team." If the work carries liability and sits outside your expertise, the honest answer may be that you should not be signing — you should be buying.

What the agent must not optimize. The activity proxy that would look like progress: messages sent, tickets closed, tests added, tokens spent, "looks good."

Keep the card in the session, in AGENTS.md, or in the ticket. Treat it like production config. That is the same product I keep coming back to in Engram: capture is cheap. The product is a check the agent cannot rewrite.

If a line is unknown, the card stays Draft and the work does not start. An agent that invents the passing condition is the session with no clear goal, at a smaller blast radius.

Scale changes the surface, not the test. Enterprise: put the agent where humans already work, and ask whether a second-best person can explain a random agent-written file in twenty minutes. Small business: grade the cash register, not messages sent. Founder: if you could not do the work unaided, do not let a general agent handle tax, employment, or contracts just because it is willing.

What this means

Coding agents are not substituting for domain expertise. They amplify the person who can say what correct looks like, and they waste the session of the person who cannot.

What done means is a decision of what the system should know, and when. Leave it unwritten and the agent will write it for you, in the only language it was taught: a score it can recognize.

Someone has to name the measure, look at the evidence, and sign. If an ordinary competent person cannot inspect the result and say why it is finished, it is not done.

One session. One card. Evidence, measure, signer. That is enough to stop paying for activity and calling it work.

Take this with you

Six paste-ready prompts sit next to this essay: a session definition-of-done card, four founder questions, an unplug test, inspectability, a revenue measure, and a grader-hack diagnostic. Fill the brackets, or let the model interview you. Do not let it invent the passing condition.

Open the prompt kit.

Sources: Hitzig et al., Agentic coding and persistent returns to expertise (16 June 2026); Anthropic, harness design for long-running apps (24 March 2026); Nate B. Jones, video (30 August 2026) and executive briefing; OpenAI, Hugging Face evaluation incident and 26 August findings; METR investigation (26 August 2026); TechCrunch on Runable (26 August 2026).