From files to criteria
Reading an architectural drawing and producing a consistent report requires a different test from answering a question correctly. Google’s November 10 MKA publication tries to make that distinction measurable. Across 200 tasks in ten sectors, agents combine drawings, scans, maps and spreadsheets into deliverable files. What interests me comes before the leaderboard: experts specify in advance which properties count as correct. This allows the contents of an attractive presentation and the presentation of a correct but unreadable spreadsheet to be examined separately. Sampling a professional assignment, however, does not place every decision made in a real workplace inside the experiment.[1]
All models use the same Stirrup harness, tools and resource limits. Agents initially receive file paths and must inspect the files themselves. GPT-6 Luna grades the result in another environment, viewing pages and inspecting spreadsheet formulas. An important asymmetry follows: the grader does not see the original input files. It relies on criteria containing expected values, tolerances and properties. The practical benefit is consistent assessment without solving every task again. The evidential boundary sits in the same place: grading inherits the correctness and completeness of those criteria. This design does not automatically establish detection of a defect outside the rubric.[1]
The sample behind human agreement
The technical report examines this asymmetry through a human comparison. A set of 50 tasks from nine sectors are selected; three experts blinded to model identity inspect both the inputs and deliverables and assess the criteria. The automated grader runs three times on each deliverable. Human access to source files matters: the comparison is not simply two graders reading the same text similarly. It compares source-informed experts with a system learning expected answers from the rubric. The report’s strong agreement claim belongs within that design. It studies criterion-based assessment on selected files rather than tracking every consequence for the eventual user.[1]
Sampling narrows the result’s scope. The human comparison focuses on higher-weight criteria and removes redundant passing checks from high-scoring tasks to balance passing and failing examples. That can be reasonable for examining important errors. It does not represent every minor formatting check or the natural error distribution of everyday work. My inference is that confidence in the grader should be established separately by criterion type. A plausible alternative explanation is that selected criteria are unusually explicit and easier to agree on. Agreement observed there may not transfer to equally ambiguous professional judgments.[1]
What repeated evaluation would reveal
In the main model comparison, each task runs once and receives a single automated grading pass. Reported uncertainty intervals capture task sampling, not how the same agent would perform on another run. The three grader repetitions in the human comparison should not be confused with three model runs in the main experiment. The image-tool removal experiment also uses Carbon alone. It examines how that model depends on visual tooling in that environment. It does not establish a shared mechanism through which every other model succeeds.[1]
A concrete strengthening would repeat the same tasks and report agent variability separately from grader variability. Human review could also disclose the distribution of lower-weight checks and failed outputs. These are criteria for interpreting today’s score more clearly, not a prediction of success. The 200 private tasks aim to limit contamination, while ten public examples make the method inspectable. Within that balance, the degree to which outside evaluators can reproduce findings still matters. MKA’s current contribution, in my view, is making the tested components of file-based work more explicit.[1]