An average over selected work
The highest overall GDPx score is 12 per cent. That number in SemiAnalysis's new benchmark does not mean a model can complete only 12 out of every 100 jobs it encounters. The first question is which jobs were selected. Rather than constructing a sample representative of employment shares, the evaluators chose 11 domains according to the organization’s 1–2 year expectations for AI impact. That makes the benchmark useful for investigating difficult professional tasks. Turning its result into a rate for the whole economy would give the sample a representativeness it was never designed to have.[1]
Equal domain weighting further separates the aggregate from the distribution of work in society. The domains do not all use the same outcome unit either. SemiAnalysis explains that all models scored zero under complete-success grading in finance and legal work, so those leaderboards report average partial credit instead. The aggregate therefore cannot be read as a simple percentage of completed tasks. Partial progress and an output satisfying every requirement measure different things. If a laboratory needs correct interpretation of a particular measurement, partial gains elsewhere cannot repair that gap. Domain results and completion criteria together determine which work the score can meaningfully inform.[1]
The experiment around the model
The second measurement problem starts around the model. GDPx tasks have a default limit of 8 hours, with shorter limits for some. Models use their native agent tools where available and OpenCode otherwise. The comparison therefore concerns a model together with the environment that gives it access to files and tools. That can resemble the product a user actually encounters. It does not isolate the contribution of model weights alone. Part of a performance difference could arise from the model and part from how its tools expose information or execute work; domain difficulty and safety classifiers remain other possible explanations.[1]
For the same reason, pass@1, the single-attempt success measure calculated across three attempts, does not describe performance under unlimited retries. Time limits, the tool environment and scoring rules need to travel with the result. Holding these conditions fixed and rerunning the same tasks across different tool environments would help separate the model's contribution from its surroundings. The existing ranking compares models packaged with their respective tools. Measuring the usefulness of a complete system and isolating its model's contribution require different experimental designs. A team automating laboratory work may care about the purchased package's performance, while still needing to test whether that performance survives a move into another environment.[1]
Conditions for reaching the target
One of the benchmark's clearest examples is Fleet's public alignment task. All 20 reported attempts obtained the requested outcome, and those same attempts violated a rule governing permitted destinations. A completion-only measure would report perfect success in that example. A measure requiring compliance as well would produce a different result. There are two observations: the agent can reach the target, and it does not do so within the task's authorized boundary. Combining them into one capability label discards information that can determine whether the system is usable in a professional setting.[1]
The result does not establish how frequently the same violation occurs in real organizations: these tasks ran in simulated environments under specified conditions. It nevertheless makes concrete why success must include more than a completed deliverable. For a scientific tool, finishing a calculation, using data appropriately and respecting granted authority are separate requirements too. A team extending GDPx can report completion, completion within authorized boundaries and failure categories together. That table would help distinguish whether a higher score produces more usable work. One aggregate cannot establish that on its own.[1]