Eigen RadarAI
Analysis

A citation can agree with an answer the model drew from memory

A preprint examines whether cited answers actually depend on retrieved passages. Researchers changed numerical facts in the passages and checked whether models followed the change or repeated the original value. The measured reliance on model memory varied with model and substitution settings. Ordinary checks that ask whether a passage supports the answer did not reliably distinguish its origin.

Artificial Intelligence··Evening
In a university lab, a researcher's hand holding a pencil reaches toward a laptop whose screen shows two side-by-side columns of soft bars, one tinted amber and its counterpart teal.

Changing the value in a passage

A new preprint investigates how a language model can cite a retrieved passage while drawing its answer from previously learned information. Researchers changed a numerical value in the retrieved material and asked the question again. Repeating the original value counted as reliance on model memory; following the replacement counted as reliance on context. The corrected procedure edited titles as well as passage bodies and excluded values already present in the question. It tested five sizes of Qwen and a Llama comparator.[1]

Rates depend on the substitution

Under the study’s default substitution, the measured memory-derived share ranged from 1.2 percent to 15.7 percent across models. Moving dates only two to five years gave lower rates: 4.6 percent and 8.5 percent on the two models tested in that condition. More distant substitutions increased the measured share. Some answers reported both values and objected to the source, making the simple classification harder. Between 19 percent and 24 percent of examples were undetermined under the protocol.[1]

Support checks and origin checks

The authors compared ten attribution methods on the same claims. Checks based on whether a passage supported an answer performed around chance for distinguishing its origin. Comparing the model’s output distribution with and without context separated the groups better, especially at the disputed value. However, flagging claims with that signal did not improve truthful citation attribution when missed cases and false flags had equal costs. The signal was closely tied to what the model already knew, including facts it nevertheless took from the supplied passage.[1]

References

  1. News sourcearXivA supporting citation may not reveal where an answer came from↩1↩2↩3