Artificial Analysis has revised its Decision Index to score how confident a model is as well as whether it chooses correctly. The evaluation adds document tasks requiring every answer to be right and tests consistency when questions are reformulated. Private datasets now have a larger weighting, while changes to the formula and tests alter the basis of comparison.
AI evaluator Artificial Analysis released Decision Index v2 on 9 November, revising how it grades models that choose between fixed options. The new Decision Score combines correctness with the probability assigned to the chosen answer. A confident correct answer scores more highly, while confidence in a wrong answer lowers the result. Questions receive scores between zero and one hundred, with correct answers above 50 and incorrect answers below 50.[1], [2]
The new AA-Paperclip test has a 30 per cent index weighting and contains 1,500 documents, including resumes, receipts, invoices and AI research papers. Models extract information and answer decision questions about each document. All 15 responses must be correct for the document to count as successful. AA-Symmetry, weighted at 15 per cent, uses 120 questions to check whether probabilities stay consistent when equivalent questions are reformulated.[1]
Private datasets now provide 45 per cent of the index, up from 25 per cent. WildGuardTest was removed because the evaluator considered it saturated. AA-Omniscience-Decision and HLE-MultiChoice each have a 15 per cent weighting, while When2Call has 10 per cent. The formula, test composition and weights have all changed, so version two uses a different measurement basis from its predecessor.[1]
