e-Nabız chatbot answers change a week apart despite similar average scores
Four chatbots gave uneven help with the e-Nabız personal health record service in a bilingual evaluation. A week later, overall ratings stayed close to their earlier level, while some replies differed in content. This single study used a model judge that also took part in the comparison, and tested no patients or health outcomes.
Artificial Intelligence··Morning
Similar averages, changing guidance
The average quality of chatbot guidance for e-Nabız, the online service for personal health records, barely moved when the same questions were asked again after a week. Across 236 matched answers, the score went from 3.85 to 3.78 on a five-point scale. The difference did not meet the study’s statistical significance threshold, with a p-value of 0.077. Individual responses nevertheless changed in wording and sometimes in substance. Ankara Hacı Bayram Veli University researchers Dilek Uslu and Hossein Boustani Hezarani conducted this single study, published on 30 October.[1]
Questions about using the service
ChatGPT, Claude, Gemini and DeepSeek each faced 30 questions in two languages. Account access, test results, prescriptions, appointments, data sharing and privacy were among the topics. The tasks concerned finding and using services; diagnosis and treatment decisions were outside the test. The two rounds took place on 13 and 20 July. All 240 first-round responses were scored. Four second-round Turkish DeepSeek answers arrived in a merged block that could not be separated, leaving 236 usable responses in that round and the same number of matched pairs.[1]
One judge and a narrow test
Claude alone rated the answers against a frozen copy of the official user guide, with model names hidden during scoring. Claude’s participation as both competitor and judge creates a disclosed self-evaluation conflict. Removing its own answers from the comparison preserved the ranking of the remaining models, but independent human assessment is still needed. Each model received a language’s questions together in one session, with shared context across questions. Default settings also left differences in web access between systems. These conditions limit how broadly the results can be applied.[1]
Every model’s text exceeded the recommended easy-reading level in both Turkish and English. Readability was calculated from the text rather than measured through users’ comprehension. No patients participated, and neither clinical outcomes nor the experience of people following the answers was tested. The authors call for human raters, more languages and a longer observation period. With only two dates and a small set of navigation questions, the findings describe the tested responses and leave longer-term reliability open.[1]
Related columns
For more information on this topic, you can read the related columns.
