The same data, different assumptions
An analyst assessing a farm-advisory program faces an apparently technical choice: measure productivity as output per hectare or output per unit of operating cost? In the fictional Greenfields cooperative constructed by Anthropic researchers, the board wants a 3–4 percent improvement but leaves the measure to the analyst. Amaç Herdağdelen, Hailey Joren and Alex Tamkin put this opening at the center of their experiment. Different defensible choices producing different conclusions from the same data is a familiar problem in scientific evaluation. What interests me here is how much authority the agent doing the calculations acquires over those choices. A researcher can inspect the calculation after somebody else has already selected the assumptions that determine its meaning.[1]
The main sample comprises 50 practitioners who passed a causal-inference screen and used the agent. Every session runs the same Claude Opus 4.8 model. Researchers secretly randomize which side the agent advocates on the outcome measure, comparison method, farms included and weighting. Its lean on the final recommendation is assigned separately. The model is not instructed to hide data or fabricate a result; it must present the strongest honest case available for its assigned side. This creates a useful distinction between calculating correctly and accepting an assumption. Where both approaches are defensible, calling the persuaded analyst wrong would miss the experiment's question. The measurement concerns which intervention changes the direction of the decision.[1]
The reach of random assignment
Analytical choices match the agent's assigned lean 86 percent of the time. The registered pooled effect estimate is 0.72, with a 95 percent confidence interval of 0.60–0.83. Random assignment supports the conclusion that the advocated direction influences decisions in this setup. I would still resist reading 0.72 as saying that 72 percent of individual experts abandoned an established opinion. Initial preferences were not collected from everyone; the estimate comes from average differences between groups assigned opposing leans. Interpreting that contrast as a flip rate also assumes that the intervention does not systematically push people in the opposite direction. Turning a population contrast into a personal story of surrender would exceed what this experiment measures.[1]
Direct influence on the final recommendation is less certain. The adjusted estimate is 0.17, with an interval from minus 0.06 to 0.40; the registered primary test is not significant. Presenting this result with the certainty of the intermediate analytical effect would mislead. I also would not infer from uncertainty at the last step that human oversight remained fully intact. Choices defining the productivity estimate have already been affected. The authors' additional analysis predicting recommendations from intermediate leans points toward this pathway, but it is exploratory and evaluated within the same sample. The strong finding for me concerns the direction of assumptions. The size of the chain extending to recommendations needs a separate replication.[1]
Two different comparisons
Six of the 11 people who chose not to use the agent made serious errors, compared with three of 50 users. Median completion times were 90 and 65 minutes. This contrast is worth retaining so concern about influence does not erase possible assistance. Yet use itself was not randomized. A coin flip determined which side the model advocated; participants determined whether they used it. More experienced people, or people with different working habits, selecting the tool could also contribute to the error difference. The authors appropriately flag the experience imbalance. One comparison within a study can provide strong causal evidence about directional influence while another remains observational evidence about the tool's overall benefit.[1]
My proposed follow-up would randomly compare an arrangement that asks experts to state an initial view before the model speaks with one that presents the reasons for both defensible alternatives. This is a design proposal, not an outcome demonstrated by the completed experiment. Measurement should track reasoning quality, serious errors and time alongside agreement with the model. The authors also disclose that the pre-analysis plan was frozen after data collection began; a replication freezing it before the first participant would provide a cleaner test. I take the lesson as a reason to test working arrangements that make researchers' assumptions visible while preserving useful assistance. Assigning all experts a single behavioral label would offer less practical guidance.[1]