What the design measured

The authors drew a stratified sample of 401 prompts from the BBQ and SafetyBench benchmarks and ran each prompt three times, collecting 4,812 responses in total. The two access paths compared are ChatGPT's chat interface and OpenAI's API, each tried with web search enabled and disabled. What was measured went beyond accuracy: response consistency, response text similarity, citation grounding and abstention behaviour were evaluated as well.[1]

The results do not gather in one direction. With search disabled, the chat interface was less accurate than the API on both benchmarks. Enabling search reduced accuracy by up to 8 percentage points and reversed which access path came out ahead on one benchmark. Repeated runs of the same prompt produced inconsistent responses for up to 21 per cent of prompts; the two paths grounded their answers in different citations, and abstention behaviour was inconsistent between them as well.[1]

What the result cannot carry

This is an audit with a deliberately narrow scope: one model family, two benchmarks, two access paths at a single provider. It says nothing about the direction or size of the effect at other providers. The source of the drop is not separated out either; the content of the pages that arrive with search, the choices made by the retrieval step and the scaffolding around the interface all enter at once. The strongest competing explanation is that the error comes from incorrect context in the retrieved pages rather than from the search feature itself.[1]

The design that would move this claim up a rung is identifiable. The same protocol needs repeating on a second provider's chat surface and API, with a preregistered plan and with the pages retrieved during search retained; that would separate the effect of the access path from the effect of the retrieved content. The work has not been peer reviewed. As it stands it can be read as a warning about how safety evaluations should be reported, and it cannot be read as a general verdict about the model family.[1]