Returning to the same question
Thirty e-Nabız questions, two languages, four chatbots and two rounds a week apart: Hezarani and Uslu ask whether the guidance a patient encounters stays consistent over time. In responses collected in July, mean quality moved from 3.85 to 3.78, with p=0.077. That result alone does not establish that asking again produces reliably equivalent guidance. An average can combine an improvement on one question with deterioration on another. A patient encounters an individual answer, not the average of thirty questions.[1]
The useful distinction is that the study separately measures aggregate quality, item-level score agreement and wording similarity. Weighted kappa is 0.73 for quality and 0.57 for accuracy; mean word-frequency cosine similarity is 0.48. Cosine similarity is not a semantic error rate. Correct guidance can be expressed in different words. The paper also describes substantive changes, including an incorrect shift in the age limit for family access between rounds. Evaluation needs to catch altered procedural guidance without treating every paraphrase as an error.[1]
Turkish Claude fell from 4.73 to 3.93 on overall quality, but attributing the entire change to a model becoming worse over a week would overreach. Thirty questions in each language were batched into one session, default settings were retained and web access was not equalized across systems. Sampling variation or interactions among questions within the session could also explain the difference. Two dates do not establish a lasting direction of change. What is observed is a problem with recovering equivalent guidance under these particular conditions.[1]
Who scores, and which answers are left out?
The scores have a second limitation: Claude is both an evaluated system and the sole judge. The researchers withheld model names and anchored scoring to the official guide and explicit criteria. Removing Claude from the comparison preserved the ordering among the other three models. That usefully checks whether the comparison depends entirely on Claude scoring itself, but it does not replace independent human validation: the same judge still scored the remaining answers. Hiding names does not automatically eliminate an evaluator’s preferences for style or content.[1]
Missing answers also belong in an assessment of usability. Four Turkish DeepSeek answers in the second round arrived in one inseparable block and were excluded, leaving 236 matched pairs instead of 240. That is understandable for scoring individual answers. Still, accuracy among separable responses and the share of all requests producing usable responses should be reported separately. In an interface, failure to obtain understandable steps is more than a missing observation in a statistical table.[1]
My design proposal is to repeat each question several times in independent sessions and have human raters assess whether the correct procedural steps are preserved. An overall quality score can remain alongside that measure; it cannot alone tell us whether the information needed to finish a task has changed. This small experiment, scored against the official e-Nabız guide, also does not measure whether real patients understood the text and completed the operation. A further step would combine blinded human assessment with user task-completion trials. The present results cannot establish clinical safety or a general ranking of current models.[1]