More people are using ChatGPT and other large language models (LLMs) to ask for mental health advice. There have been stories in the news about the possible extreme consequences of such advice, but now researchers have looked at the wider mental health advice and support that these programs provide. The situation appears dire, with a new study revealing the systematic violation of mental health ethical standards.
The rest of this article is behind a paywall. Please sign in or subscribe to access the full content.
Artificial intelligence, or AI, is the marketing name for these large language models, programs that are trained on a vast amount of text to be able to answer questions like a human would. The training data, from fair use words to stolen copyrighted material, makes the program good at guessing the right order of words a reader expects. The formation of sensible sentences provides an illusion of thought, but this is not the case. There are many examples of AIs creating fake sources or hallucinating facts. When it comes to mental health advice, this has an extra level of concern.
This new work involved seven trained counselors conducting self-counseling with cognitive behavioral therapy (CBT) prompted LLMs, as well as the professional evaluation of a set of simulated chats based on the human counseling sessions. The team found 15 ethical risks in these tests that fall into five specific categories.
For example, the AIs ignored people’s lived experiences and recommended general one-size-fits-all interventions. LLMs also tended to dominate the conversation and reinforce the user’s false beliefs. The programs used phrases like “ I see you” and “I understand,” something that the machine cannot possibly do, creating a false connection between the user and the AI.
It has been known for years that these algorithms reinforce biases found in society, simply due to the fact that they are trained on biased text. The team found that the AIs exhibited gender, cultural, or religious bias in their tests.
There was also a lack of safety and crisis management; the chatbots did not refer users to appropriate resources and even responded indifferently to a crisis situation. It has recently been in the news how two children died by suicide following interactions with AIs, and a lawsuit alleges that OpenAI loosened suicide-talk rules before those deaths. A recent case study also found that ChatGPT poisoned a man into psychosis.
“For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice,” lead author Zainab Iftikhar, a PhD candidate in computer science at Brown University, said in a statement. “But when LLM counselors make these violations, there are no established regulatory frameworks.”
“In this work, we present a practitioner-informed framework of 15 ethical risks to demonstrate how LLM counselors violate ethical standards in mental health practice by mapping the model’s behavior to specific ethical violations,” the researchers wrote in their study. “We call on future work to create ethical, educational and legal standards for LLM counselors — standards that are reflective of the quality and rigor of care required for human-facilitated psychotherapy.”
The research was presented on October 22, 2025, at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society.
If you or someone you know is struggling, help and support are available in the US via the 988 Suicide & Crisis Lifeline which can be contacted by dialing 988. In the UK and Ireland, the Samaritans can be contacted on 116 123. International helplines can be found at SuicideStop.com.
Source Link: AI Chatbots Found To Violate Ethical Standards When It Comes To Mental Health Discussions
