DecisionPivot tests whether AI decisions follow the detail that changes the answer
DecisionPivot evaluates models that route requests, check policies and verify outputs using questions where a pivotal detail changes the correct answer. The preprint combines generated contrast groups with existing public tests. Its measures distinguish getting one item right, correctly answering every related variant and assigning probabilities in the right order, exposing differences hidden by aggregate scores.
Artificial Intelligence··Evening
A small change makes a different answer correct
DecisionPivot, a benchmark described in a single arXiv preprint, tests decisions used for routing requests, checking policies and verifying outputs. Its generated questions form contrast groups: a pivotal detail changes which answer is correct. One task asks whether a required file survives a workflow. Reordering its steps changes the correct answer. A solver implemented separately from the generator provides a check on those calculated answers.[1]
Generated tasks sit alongside public reference tests
The benchmark covers seven decision roles, with 40,033 items organised into 48 tasks. Its core track has 7,800 items from 13 generated task families; the reference track draws 32,233 items from public tests, using 35 sets. Reference tasks keep their original metrics, so their scores are not directly equivalent to core-track results. Evaluations cover 40 specialised decision models alongside 14 general models, using a documented protocol for each.[1]
Correct probability ordering can still miss the decision
Measures distinguish item accuracy, success across an entire contrast group, probability ordering and calibration. For Perplexity Decider, tests on 2,400 binary contrast pairs put the positive example higher in 92.3 per cent of cases. Correct classification of both examples reaches 74.1 per cent. The researchers also examine higher reasoning settings on a separate subset and empirical error rates in high-confidence answers. Even with generated examples, some overlap with a model's training data remains possible.[1]
