When the whole document is right
Fourteen correct answers about a resume can still count as failure if the fifteenth is wrong. AA-Paperclip, added to Artificial Analysis’s Decision Index v2, uses exactly that rule: all 15 answers for each of its 1,500 documents must be right. Resumes, receipts, invoices and research papers appear in the same benchmark. The endpoint differs from average accuracy across individual fields. It asks whether the complete bundle of work on a document succeeds. Understanding a model’s score starts with understanding that unit of success.[1]
That choice can be useful when one small document error breaks a downstream workflow. But the aggregate endpoint puts an extraction error and a decision error into the same failure category. Two systems with similar document scores could have different weaknesses: one might repeatedly miss a field, while another gets the decision wrong. Their matching totals would not establish matching failure modes. Conversely, when a real workflow needs every answer, high average field accuracy could give an overly favorable impression. The appropriate endpoint depends on the work the system must complete.[1]
Where confidence enters the score
The new Decision Score incorporates confidence alongside correctness. Correct answers score above 50 and incorrect answers below 50, with confidence moving the result within each range. A confidently wrong answer therefore differs from a hesitant wrong answer. The index’s composition has changed too: AA-Paperclip contributes 30 per cent and AA-Symmetry 15 per cent, while private datasets now account for 45 per cent overall, up from 25 per cent. A direct comparison with a version-one score would hide changes in both what is tested and how the answers are rewarded.[1]
Calibration matters when expressed confidence corresponds to how often an answer is correct. Yet a better aggregate confidence score does not select the appropriate acceptance threshold for every decision. The cost of a wrong decision can differ from the cost of referring a case to a person. I would not read the ranking as permission to automate a particular task. Conversely, calibrated confidence could help identify cases needing review even without increasing the amount of automation. Measuring that benefit is a deployment question distinct from the model’s position in this index.[1]
Stability and correctness
AA-Symmetry tests a different property: how much do a model’s probabilities change when a question keeps its meaning but changes its form? Its 120 questions undergo transformations including reordered options, swapped labels and negated yes-or-no formulations. That design examines sensitivity to presentation when the underlying decision should remain the same. Consistency and correctness are nevertheless different properties. A model could preserve the same wrong answer across equivalent formulations and appear stable. That is why invariance needs to be interpreted alongside correctness rather than as another name for it.[1]
The useful change is that decision quality is no longer reduced to the number of correct answers alone. Confidence and resistance to changes in presentation become visible too. The next concrete improvement I would find valuable is a breakdown of extraction and decision errors alongside the document-level result. It would help show whether a weakness lies in finding information or deciding from it. The aggregate can still be decisive when a workflow requires a perfectly completed document. Component results would not invalidate that total; they would make it easier to judge which work the result can reasonably inform.[1]