The harder question after a good answer
Imagine asking a language model to reschedule a meeting by email. A good response might encourage you to trust it with a more sensitive workplace conversation next time. Hannah Li’s new theoretical paper examines the measurement consequence of that possibility: a model can change the tasks users bring to it. Success on a common set of simple questions and success on questions users choose over time are different targets. The useful contribution, in my reading, is to make the target explicit before evaluation begins. An aggregate success rate interpreted without the difficulty of the tasks encountered can support a misleading model ranking.[1]
The email example is constructed rather than an experiment observed in real users. Under its specified probabilities, users receiving excellent responses move to harder tasks. The stronger model can perform at least as well on each task type while receiving a lower aggregate success rate, because the share of difficult tasks increases in its later requests. This does not tell me that a particular product behaves that way. It establishes how the percentage can combine model capability with task selection. Applying that explanation to a deployment would require measuring which users attempted which new tasks.[1]
What the two tests estimate
Giving both models a fixed prompt set measures a difference on a common distribution. Randomly assigning users to two models and following their conversations measures a difference on the distributions each model induces. Random assignment in the second design does not keep subsequent questions identical. Users changing direction in response to an answer become part of the experiment. When the methods disagree, my first question would therefore extend beyond sample size to whether they target the same quantity. Collecting more observations does not automatically turn different target quantities into one measure.[1]
I would report the share of each task type and performance within each type alongside the overall score. That would help examine whether a lower aggregate reflects harder requests or worse answers to comparable tasks. The paper provides a sufficient condition involving prompt-distribution shifts and differences in performance across tasks. Applying it requires meaningful task categories. Combining every writing request into one category could hide the distinction between routine rescheduling and a sensitive workplace exchange. Very fine categories can make data collection difficult. The measurement should be interpreted with sensitivity to how those categories are chosen.[1]
Choosing the experiment horizon
A longer experiment can look attractive because users may need time to learn what a new model can do. The paper raises another measurement issue here. Dropping an initial adjustment period puts more weight on later turns, when the models’ prompt distributions may have diverged further. That can suit a long-run usage objective. A comparison of answer quality on the same tasks would instead be moving toward a different question. I would choose the relevant usage horizon before inspecting the success table. Reporting short- and longer-term outcomes together could also make adaptation easier to see.[1]
The theoretical simplifications also matter. Main results let the next prompt depend on the preceding response and the model response on the current prompt, while assuming independence between users. A simpler linear example illustrates how feedback can make successive observations dependent and increase estimation variance. It supplies no ready-made number for how many additional users a real service needs. My proposed next test would jointly measure task choice, changes within users over time and task-specific outcomes in real interactions. That would let the harder-questions explanation be compared with other possible behavioral changes in the same data.[1]