A citation is not a receipt.

A working link can support the wrong sentence

An AI answer looks more serious when it has footnotes. The sources are real. The links open. The formatting is immaculate. Unfortunately, the paper describes a different population, the policy expired last year, and the paragraph makes a claim none of the cited pages actually makes.

This is not just a hallucinated-URL problem. An answer can retrieve a genuine document and still use it badly. Retrieval establishes that material was found. Citation establishes that the answer points somewhere. Evidence requires a defensible relationship between a particular claim and the relevant part of that material.

The ALCE research benchmark separates answer correctness from citation quality, alongside fluency. That distinction is useful when evaluating a research assistant: an answer can read well while its evidence is incomplete, and a correct statement can still be inadequately supported. The paper’s tasks and historical model results should not be treated as a performance estimate for a current product. ^[1]

Our proposed audit starts with claims rather than links. It is designed for teams deciding whether an answer is useful enough to publish or act on, not for producing a reassuring percentage next to a bibliography.

Break the answer into claims someone could check

Take this invented sentence: “The new system reduces processing time by 30%, costs less, and works reliably for small businesses.” It contains at least three claims. A source showing a processing-time result under a particular benchmark does not automatically support the cost comparison or the small-business generalization.

Create one row per checkable claim. Record the exact wording, the cited source, the relevant passage, and the qualifications needed to interpret it. Those qualifications often include the date, population, comparator, measurement, and operating conditions. A result measured against one baseline should not quietly become a claim about every alternative.

Do not require a citation for every connective phrase or explicitly stated opinion. The point is to identify claims that a reader might reasonably rely on as facts. “We would test the smaller model first” is a recommendation. “The smaller model is more accurate on this task” is a comparative claim that needs evidence about that task.

Mark inference separately. A source may establish that a tool supports a certain input type; your team may infer that it could fit a particular workflow. Both can belong in the article, provided the transition is visible. Making the inference sound like a documented product guarantee is where a useful analysis becomes misleading.

Check support before checking prestige

A famous source is not a universal permission slip. Ask whether the cited passage supports the claim at the same level of specificity. If the answer says “all users” and the study says “participants in this experiment,” the answer has expanded the scope. If it says “caused” and the source reports an association, it has strengthened the relationship.

Use a small review vocabulary: supported, partly supported, contradicted, or unresolved. “Unresolved” matters. A paywalled passage you cannot inspect is not automatically false, but it also should not pass because the title sounds relevant. A document that changed since the answer was generated needs its version recorded before you can judge what the model saw.

For partly supported claims, identify the missing qualification and repair the sentence. A useful correction might be as small as adding the benchmark name or as substantial as replacing a universal recommendation with a description of a limited experiment. The aim is accurate communication, not a citation-density competition.

Where several sources jointly support a claim, record what each contributes. Do not let a pile of links conceal that none supports the central assertion. Conversely, one precise primary source can be better than five derivative summaries repeating it.

Two citation scores can tell very different stories

Consider a hypothetical audit of twenty answers containing eighty checkable claims. Sixty claims have citations. Reviewers find that forty-eight of those cited claims are supported, while twelve are not fully supported. Twenty claims have no citation.

For this particular claim-level audit, supported cited claims divided by all cited claims gives 48 ÷ 60, or 80%. Supported cited claims divided by all checkable claims gives 48 ÷ 80, or 60%. We can call these supported-citation precision and supported-claim coverage, provided we publish the definitions. They are our simple audit measures, not a reproduction of ALCE’s complete evaluation procedure.

The first number says the cited portion is often defensible. The second says a substantial part of the answer set still lacks verified support. Neither is an answer-accuracy score. Some uncited claims may be correct. Some answers may omit a crucial fact while making only well-supported statements. A source can also be faithfully cited and itself be wrong or unsuitable for the decision.

Count broken links separately. If all eighty claims have functioning URLs, that would be a perfect retrieval-access score under one definition and still tell us almost nothing about whether the conclusions follow. This is why a citation audit should retain the individual claim decisions instead of collapsing everything into one green badge.

For high-impact claims, the practical release rule may be stricter than an aggregate threshold: a wrong price, fabricated capability, or reversed finding blocks the answer even if the other nineteen answers look good.

Make the evaluation difficult in the right way

A useful test set includes questions your system is likely to mishandle: an old policy beside its replacement, two products with similar names, a benchmark whose headline hides a narrow setting, and a question the available documents do not answer. The last case tests whether the system can acknowledge a gap instead of filling it with confidence.

Keep a small source snapshot or version record for each evaluation item. Record the question, retrieved passages, answer, and citation mapping. If the retrieval corpus changes between comparisons, a better answer might come from better documents rather than a better model. That may still be a product improvement, but it is a different explanation.

Have reviewers assess answers without knowing which model produced them where practical. Start with a shared rubric and review a sample together. For important or ambiguous claims, use a second reviewer and document disagreements before resolving them. Agreement is evidence that the rubric is understandable; it does not prove the reviewers are correct.

Report results by useful failure type. Missing evidence, overgeneralized evidence, stale evidence, and a citation attached to the wrong clause suggest different fixes. Increasing the number of retrieved passages is unlikely to solve all four. It can also make the model’s evidence selection more difficult.

Fix the smallest broken part of the evidence chain

If the source is absent, improve retrieval or admit the gap. If the source is present but the claim overreaches, revise the answer instructions and examples. If citations are attached after the answer is written, test a workflow that records supporting passages while drafting. If documents disagree, surface the disagreement rather than selecting whichever produces the cleanest sentence.

Avoid making the same model’s approval the only acceptance test for its own work. A second model can help flag mismatches, but that is another fallible component to evaluate. Preserve an inspectable trail from each consequential claim to the relevant evidence so a person can resolve uncertainty quickly.

The AI workflow evaluator can help frame the broader deployment decision: what job the system performs, what a failure costs, and which checks belong before release. A research assistant that is useful for finding candidate sources may not be ready to publish conclusions without review.

The best outcome is not an answer with more superscript numbers. It is an answer whose factual claims survive being read next to the material they cite—and whose limits remain visible when the evidence stops.