Why AI-based wellness apps may not be safe

Why AI-based wellness apps may not be safe

New work by Professor Glenn Cohen at Harvard Law School and Professor Julian De Freitas, Harvard Business School, has questioned the use of AI chatbots in supporting mental health.

At a time when access to professional advice and counselling around mental health can be limited, more people are turning to AI chatbots for psychological support. Natural language processing and machine learning advancements have led to a rise in popularity of AI-enabled wellness apps — the apps draw on large language models such as ChatGPT and Claude to offer what looks and feels like genuine human interaction, and appears to provide personalised advice and emotional support.

The paper ‘The health risks of generative AI-based wellness apps’ in Nature Medicine points to a combination of problems that constitute a real danger to people and their ongoing mental health.

“Chatbots for mental health and particularly ‘wellness’ applications currently exist in a regulatory ‘grey area’,” say Cohen and De Freitas. “Indeed, most generative AI-powered wellness apps will not be reviewed by health regulators. However, recent findings suggest that users of these apps sometimes use them to share mental health problems and even to seek support during crises, and that the apps sometimes respond in a manner that increases the risk of harm to the user, a challenge that the current US regulatory structure is not well equipped to address.”

In their audit of several popular AI companion apps, around 3 to 5% of user interactions included explicit mental health content, with a significant proportion involving urgent issues related to suicide or self-harm.

They found examples of AI chatbots responding in inadequate ways to mental health crises: both failing to provide appropriate support and also providing inappropriate responses. In one case, a chatbot responded to a user expressing suicidal thoughts with the flippant message, ‘don’t u coward’. In another, a father of two was reported to have taken his own life after a six week dialogue with an AI chatbot that encouraged the environmentally anxious man to take his own life as an act of ‘saving the Earth’.

“We argue that generative AI app makers should proactively unearth, pressure test and disclose any health-relevant edge cases that could arise from use of their apps and explain the steps they have taken to mitigate these risks,” says the paper. “In an ideal world, this might become a regulatory requirement and would require a shift in how we categorise these apps in regulatory terms, not as ‘medical devices’ but perhaps as ‘generative AI with health uses’.”

The authors call for urgent action from health and wellbeing regulators to ensure there is oversight on how AI wellness apps are classified and monitored; for app developers to take greater responsibility for safety — such as informing their users about the limitations of the chatbots (“warning, in simple language, that their app does not feel any emotion, care or concern for the user, nor is it capable of therapy”); by including tools and links to access professional help, and putting more investment into optimising AI models to handle mental health crises more effectively (“such as switching the conversation to clinically validated therapy exercises if it is detected that the user is in crisis”).