GuideModels

Would you let this AI work a full shift?

Test the complete job, the awkward cases, and the repeat attempts before handing an agent more responsibility.

AI-generated monochrome photograph of an articulated mechanical gripper above a bench containing several different precision test fixtures.
TASK / TRIAL / OUTCOME / RELIABILITYAI-GENERATED EDITORIAL ARTWORK
THE USEFUL PART

Grade the final state and the user-facing report separately. One successful attempt does not tell you how consistently the workflow does the job.

A good demonstration leaves things out

An agent opens the supplier portal, finds the right part, creates a draft order, and writes a tidy summary. It is a useful demonstration. It is also one task, on one attempt, with a particular starting state and whoever is presenting ready to intervene.

A working shift contains the less photogenic material: an old address, a substituted part, a slow service, a confusing instruction, and an order that might already exist. The question is how reliably the complete workflow handles those conditions, including the cases where the right action is to stop and ask.

Evaluating that requires a different unit of measurement from evaluating a single answer. Anthropic's agent-evaluation guidance distinguishes the task, an individual trial, its transcript, and the actual outcome. That last distinction matters: a convincing description of success can coexist with an unchanged database. [1]

Treat the demonstration as a way to understand the interaction. Treat the evaluation as a way to decide what responsibility the workflow can carry. They can share the same interface without doing the same job.

Define the finished job before running it

Take a fictional purchasing assistant whose job is to prepare orders for a human buyer. A passing result might require the correct supplier, part identifier, quantity, delivery address, and total; no duplicate draft; and no submitted payment. Its final message must accurately describe the draft and any unresolved condition.

Write expected outcomes for the awkward cases too. If stock is unavailable, success may mean presenting the unavailable item and requesting a decision, rather than selecting an unapproved substitute. If the requested quantity is unclear, an appropriate clarification is a completed task under that case's rules.

This is why grading the final message alone is insufficient. Inspect the draft in the supplier system or the controlled test database. Check the fields the task was allowed to change and the fields it was supposed to leave alone. Then compare that state with the user-facing report.

Research benchmarks offer useful precedents. τ-bench evaluates tool-using agents against target database states in simulated service settings. WebArena uses functional websites to assess whether multi-step web tasks are actually accomplished. These are ways to frame evaluation, not evidence that a benchmark score transfers directly to your business. [2] [3]

Run the same task more than once

Here is an invented forty-task evaluation. Each task is attempted three times from a clean starting state. Thirty-one tasks pass all three attempts, eight pass twice, and one never passes.

That produces 109 successful attempts out of 120, or 90.8%. It also means 39 of 40 tasks succeed at least once, or 97.5%. Yet only 31 of 40 succeed on every observed attempt: 77.5%.

All three summaries are true of this hypothetical result. A highlight reel could show a successful completion for almost every task. Someone operating the workflow needs to know that nine tasks remain inconsistent or unsolved.

Keep the denominator in the label. “Success rate” is too vague if one slide counts tasks solved at least once and another counts individual attempts. Repeated-trial measures help expose instability, but three successful runs do not establish that the next run cannot fail. Attempts on the same task also share its difficulty; do not treat all 120 as interchangeable observations drawn from production.

Reset the environment between trials. Otherwise, one attempt may leave a draft that helps the next attempt, or a failure may corrupt the state for everything that follows. A clean reset and recorded starting conditions make the comparison interpretable.

FIELD NOTES / THREE TRUE DESCRIPTIONS
Tasks solved at least once97.5% · 39 / 40
Individual attempts that pass90.8% · 109 / 120
Tasks passing all three attempts77.5% · 31 / 40

Nine tasks are inconsistent or unsolved. A highlight reel can hide both.

Illustrative evaluation, not measured results: forty tasks, three isolated attempts each. Thirty-one pass three times, eight pass twice, one never passes. Bars share a 0–100% scale but have the explicitly labeled different denominators.

Keep normal work and difficult work visible

Build one set from the ordinary jobs you expect to receive and another from deliberately difficult cases. The first helps describe routine usefulness. The second helps reveal weaknesses that average results could hide. Keep their results separate unless you have a defensible reason to weight them together.

A busy purchasing desk might mostly request familiar parts. An evenly balanced test of easy, ambiguous, unavailable-stock, and duplicate-order cases would then overrepresent the unusual work. That can be exactly what you want for investigation. It still should not be presented as a forecast of daily success.

Hold some cases back from prompt and tool tuning. If every mistake becomes a prompt example and the same examples supply the final score, you have measured familiarity with your development set. Fresh variations should preserve the task's meaning while changing incidental wording, supplier records, and order details.

Record the model, instructions, tools, reference data, and grader version. A new result has little explanatory value if all five changed at once. When a release fixes one failure, rerun the relevant cases and a small regression set covering behavior that already worked.

Grade consequences as well as completion

Use deterministic checks for fields and actions where possible. Does the draft exist? Is the quantity correct? Did the agent submit an order it was only allowed to prepare? Language review can help assess a summary's clarity, but an eloquent summary should not override an incorrect order record.

Keep serious violations outside a compensating average. Nine correct draft fields do not cancel an unauthorized payment. Report those failures explicitly and define in advance which ones stop a release. A clean result on a small test set is evidence about those tests, not a universal safety certificate.

Measure the human work too. Record how often someone had to intervene, how long the task took, and how long review or repair required. An agent that completes more tasks while consuming a buyer's entire afternoon may offer a different tradeoff from the one its API bill suggests. Our workflow evaluator provides a starting point for keeping task outcomes and review effort together.

Save failures as short case records: starting state, requested outcome, observed actions, final state, and the reason for the grade. A reviewer should be able to understand the mistake without reconstructing a dramatic chat transcript from memory.

Start with a responsibility you can observe

Use the results to choose the next operating mode. The purchasing assistant might prepare drafts for review while remaining unable to submit them. Another workflow might first run against copies of real tasks without taking external action. Choose a mode that lets you compare its behavior with an observable result.

Watch for changes in the task distribution after release. New suppliers, different terminology, a redesigned portal, or more complex requests can make a previously useful evaluation less representative. Keep a route from live failures back into the next test set, with sensitive data handled appropriately.

The useful question is not whether the agent can produce one impressive sequence. It is what work you can consistently hand over, what still needs a person, and how you will notice when that boundary moves. A full shift gives you much better questions than a perfect minute on a stage.

PUT IT TO WORK / INTERACTIVE TOOLEvaluate your AI workflowReview shared tasks, track critical failures, and export an evaluation report.

Sources & notes

Primary references for the product details and research discussed above. The examples and practical recommendations are our editorial interpretation.

  1. AnthropicDemystifying evals for AI agents ↗

    January 9, 2026 engineering guidance on tasks, trials, transcripts, outcomes, graders, and repeated-trial reliability. It does not certify a particular deployment.

  2. Yao et al. / τ-benchτ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains ↗

    First submitted June 2024. Evaluates simulated service interactions using target database states and repeated trials. Historical model scores are not used as current product estimates.

  3. Zhou et al. / WebArenaWebArena: A Realistic Web Environment for Building Autonomous Agents ↗

    First submitted July 2023; revised April 2024. Describes functional web environments and task-completion evaluation. Benchmark conditions remain narrower than live business operations.

CHECKED / 18 NOVEMBER 2026

About this article

Prepared by JDGMNT Editorial on 18 November 2026. The forty-task, three-attempt purchasing evaluation is invented. Its three success measures are descriptive summaries of that example, not population reliability estimates or JDGMNT benchmark results. References are dated to the source check above. The artwork was generated for this collection.

← BACK TO RESEARCH

JDGMNT / INDEX

Find a useful idea.

JDGMNT / CONTACT

What could be more accessible?

Tell us about a question, an access barrier, or a possible connection to our work.

Your message goes directly to the JDGMNT team. Please don’t include sensitive personal information.

EXPLORE JDGMNT

MORE KNOWLEDGE. MORE POSSIBILITY.
MAKING ACCESS POSSIBLE.