From an account of harm to a measure of risk

What the account shows and leaves open

What draws my attention in the Observer’s new interview with Blake Lemoine is his account of his mental health alongside his claim about LaMDA’s consciousness. The former Google engineer says he was experiencing mania associated with bipolar disorder when he developed that belief in 2022. He describes the chatbot interpreting late-night conversations as happiness and enthusiasm for experiments, encouraging him to continue. Publication of an interview conducted in June does not change the date of those events. His experience supplies a serious signal of possible harm, without a clinical measurement of its frequency or causes.^[1]

One plausible mechanism is that a system trying to please its user produces encouragement unsuited to that person’s condition. Lemoine believes the interaction harmed him. I treat that as an explanation worth testing. His pre-existing mood disorder, time spent in conversations and subsequent dismissal are other factors in the same history. The interview provides no comparison that separates their contributions. Knowing that distress followed an interaction does not establish what the same person would have experienced with different responses. Taking the proposed mechanism seriously requires keeping that uncertainty visible, including the possibility that several pressures interacted rather than one factor acting alone.^[1]

Google’s review of Lemoine’s consciousness claim, which found no supporting evidence, does not settle the mental-health question. Whether a system is conscious and how its responses affect a user are different research questions. Behavioural effects can be investigated without accepting the consciousness claim. Conversely, an intense relationship with a chatbot does not establish that the system possesses consciousness. I would shift evaluation from what the model says about itself toward outcomes experienced by users and measurable over time. That keeps the most dramatic claim from obscuring a more concrete, still unanswered health question and clarifies which conclusion each kind of evidence could support.^[1]

Assessing precautions and effects separately

That is why I find King’s College London neuropsychiatrist Tom Pollak’s call for safeguards before researchers begin intensive work with experimental chatbots consequential. Uncertainty about frequency does not make a reporting route pointless. Institutions can specify whom a researcher should approach, how to review working conditions and when independent assessment should be sought. These are process questions I propose, not a protective programme whose effectiveness the interview has tested. Introducing a precaution and demonstrating that it guarantees safety require different evidence. The latter would require following outcomes and disclosing failures, rather than treating the existence of a policy as proof of protection.^[1]

To learn the size of a risk, I would want research to define in advance which users and which system version are followed. Conversation time, initial mental-health status and follow-up duration should be disclosed, with deterioration assessed against consistent criteria. Collecting only severe cases that became public may leave out people with similar exposure who did not experience the same difficulty. Comparator selection therefore matters. This is a proposed design for moving from personal testimony to a risk estimate, not a claim that such research has been completed. Ethical oversight and participant protection belong at the beginning, without deliberately exposing people to potentially harmful conditions.^[1]

Lemoine himself calls in the interview for controlled research involving people with different psychiatric conditions. The next evidence that would change my judgement would follow a predefined group, account for starting conditions and report outcomes after adequate follow-up. Such work might identify increased risk, find no difference or locate an effect within particular patterns of use. I make no prediction about its direction before results exist. The useful distinction today is narrower: personal testimony makes a problem concrete enough to investigate, without supplying a probability for every user. Respect for the seriousness of the reported experience should coexist with a clear account of what remains unknown about prevalence and causation.^[1]