What to save from a failed AI answer

Save the failure while it is still understandable

An AI answer can be wrong in several different ways. It may use the wrong source, miss a condition, invent a detail, misunderstand the task, or produce a correct answer in an unusable format. Saving only a thumbs-down loses most of the information needed to improve it.

When someone flags a problem, capture enough context to understand the failure later. Include the task, the relevant input, the output, the expected behavior, and the reason it did not meet that expectation. Remove personal or confidential information that the review does not need.

Do this while the person still remembers what they were trying to accomplish. A collection of unexplained bad answers becomes difficult to use surprisingly quickly.

Describe the expected behavior

“This answer is bad” is a reaction. “The answer recommends a product that is unavailable in the customer's country” is a failure description. “It should check availability before recommending a product” gives the team a requirement to test.

Be careful when writing a reference answer. Some tasks have one correct result, such as extracting a date from a document. Others allow several useful responses. In those cases, write acceptance criteria rather than insisting that every answer match one person's wording.

A concise record might look like this:

This is an illustrative record, not a report of a deployed system. The point is to make the problem concrete enough that another person can reproduce and evaluate it.

Look beyond the model

The model is one part of the workflow. The input may have omitted the country. Retrieval may have returned an old catalog. A tool may have failed silently. The interface may have hidden an important condition from the user.

Trace the failure to the earliest point where the correct information was lost or the wrong action became possible. Changing the prompt may help, but it will not repair a catalog that is consistently out of date.

Group failures by practical cause rather than by how embarrassing they felt. Useful categories include missing input, stale information, unsupported claim, wrong action, poor format, and unclear escalation. Keep the list short enough that reviewers use it consistently.

Turn a few failures into repeatable checks

Choose examples that represent important patterns. Preserve the conditions that caused the failure, then define what passing looks like. Record any permitted variation in the answer.

When you change the workflow, run the same checks and compare the outputs. Do not look only at the problem you intended to fix. A new instruction that improves regional availability might make the assistant refuse ordinary recommendations too often.

Keep a separate set of fresh tasks for periodic review. If the team repeatedly edits the system against the same examples, those examples become less useful as evidence that it will handle unfamiliar work.

Measure the cost of the fix

A change can improve acceptance while adding substantial review time, latency, or service cost. That may be a sensible trade, but it should be visible.

Track the number of outputs accepted under the same criteria, the time people spend checking and repairing them, and the failures that remain unacceptable regardless of the average score. Our model-comparison worksheet helps compare the cost of accepted results; it does not replace a review of serious failure types.

Avoid a single score that lets good formatting compensate for an invented fact. Some requirements should be gates. If an answer fails one of those, it fails the task even if the rest reads well.

Close the loop with the person who reported it

Tell the reporter what changed, what was checked, and what remains unresolved. If the issue was a missing input rather than a model limitation, explain how the workflow now makes that input visible.

This is useful for trust and for future reports. People give better feedback when they can see that a specific example led to a specific improvement.

Start with the next five meaningful failures. Write a clear record for each, group the causes, and choose one fix. A small collection of well-understood problems is more valuable than a dashboard full of reactions nobody has investigated.