XYBench tests whether AI answers address the user’s actual goal
Ai2 has released XYBench, an evaluation that separates a user’s requested method from their intended goal. Across practical questions, models often included a useful alternative without making it the focus of the response. The English-language benchmark measures solution presence, emphasis and recognition of mistaken assumptions separately, using forum questions and generated examples.
Artificial Intelligence··Night
A requested method can miss the goal
Ai2, an artificial intelligence research institute, announced XYBench on November 13 to examine questions built on mistaken assumptions. The evaluation separates the method a person requests from the goal they want to achieve. Its cupcake example asks for thicker frosting when cooling the cake addresses the underlying problem.[1], [2]
Ai2 describes 8,115 questions, including forum material from Stack Overflow and Stack Exchange, manually curated examples and questions generated from WikiHow guides. Highly rated and accepted forum answers helped identify intended goals. The research paper predates the announcement: it was published as a preprint on September 6.[1]
A useful answer does not always lead
Five models generated answers in the main evaluation. On Stack Overflow questions, solutions to the underlying goal appeared in 59–71 per cent of responses, but only 35–41 per cent focused primarily on that goal. Solution presence, emphasis and recognition of the mistaken assumption are scored separately. A separate multiple-choice exercise used six models, which selected goal-focused answers in 71–84 per cent of cases.[1], [2]
Human reviewers checked 500 responses
Two human reviewers assessed 500 responses alongside the automated Sonnet 4.5 judge. Reported agreement was highest for whether a solution was present and lower for answer emphasis. These agreement figures describe the grading process. The dataset covers English, mainly practical tasks, and a single response to each question; the WikiHow questions are synthetic. Long conversations and performance in other languages were outside this evaluation.[1]
Related columns
For more information on this topic, you can read the related columns.