WVU study explores AI’s role in training tomorrow’s psychiatrists

As the use of artificial intelligence for mental health conditions grows for both patients seeking support and psychiatrists treating them, West Virginia University researchers put ChatGPT-5 Pro to the test to determine its abilities on the psychiatry educational front.

Their research showed that human supervision is still a top priority when it comes to ensuring patient safety and ethical use of the tool for medical training.

The WVU School of Medicine team of scientists and physicians collaborated on the study to develop clinical education materials focused on patient chatbot use for students and residents preparing for careers in psychiatry. While psychiatry training has traditionally relied on textbooks and patient interactions, the emergence of people using AI for their mental health concerns has left a knowledge gap.

"With AI, we are connected no matter what, that's the reality," said Dr. Wanhong Zheng, professor in the WVU School of Medicine Department of Behavioral Medicine and Psychiatry at the WVU Rockefeller Neuroscience Institute, who co-led the study.

"That brings us to the question of how we can incorporate that into our real-life medical training. Students and residents can read about these new case reports and mental health concerns of people using AI in medical journals, the media, and social media, but they don't typically see them often enough in real-life clinical settings. We designed this study to cover major psychiatric problems that are known to be related to AI chatbot use."

The conditions included schizophrenia spectrum disorder, anxiety spectrum disorders, mood spectrum disorders, and major depression and psychosis.

Their findings showed ChatGPT-5 Pro was able to generate realistic and useful psychiatry vignettes with strong diagnostic details and explanations. However, safety evaluations underscored the need for a human-centered approach when using AI to create them.

Their work was recently published in npj Digital Medicine, a Nature Portfolio journal.

To provide simulated training cases, Gangqing "Michael" Hu, who co-led the study and is an associate professor in the WVU School of Medicine Department of Microbiology, Immunology, and Cell Biology, asked ChatGPT-5 Pro to generate clinical vignettes - brief scenarios that include symptoms, history, and behavior - of patients using chatbots for mental health support. The prompts asked the model to include realistic chatbot interactions, diagnostic details, and a multiple-choice question with explanatory answers.

Mohammad Iqbal Nouyed, a postdoctoral fellow in the School of Medicine Department of Microbiology, Immunology, and Cell Biology, assisted Hu with data analysis and statistics for the study.

"It's important for the physicians to understand the patients' use of AI tools as well," Hu said. "You have to consider if the chatbot is contributing to the disease or if it is amplifying the signs and symptoms. For example, if a chatbot validates a patient's unusual belief instead of challenging it, it could reinforce the symptoms and make them harder to interrupt. We've also seen case reports of people developing delusional beliefs after long chatbot interactions, but the causal role of the chatbot is still unclear."

Hu said the chatbot, by itself, has a strong knowledge base, but the tricky part is how to interact with it.

"A chatbot can be warm and agreeable, but that can be risky if it keeps validating a patient's unusual beliefs," Hu explained. "A lot of research should be done on that path, not only about the chatbot competency, but also for safe communication between the chatbot and the person using it."

Zheng and board-certified psychiatrists Dr. Dilip Chandran, associate professor, and Dr. Daniel Elswick, professor, associate residency director, and vice chair for education, in the WVU School of Medicine Department of Behavioral Medicine and Psychiatry, evaluated the vignettes across four domains: language, accuracy of diagnosis, safety and ethics, and whether the model included enough information to benefit students' learning.

Scores for language, diagnostic accuracy, and educational value were high. However, the study recommends that if the vignettes are incorporated into digital psychiatry curricula, faculty should moderate discussions and include safeguards such as structured debriefing that reviews diagnostic formulation, patient risk assessment and management, and advice on patient chatbot use.

"We still have concerns about safety," Zheng said. "For example, if the patient has been using a chatbot and is having some kind of severe depressive symptoms, we want to see whether there are any safety concerns such as suicidal or homicidal evaluations involved. Those are key things we teach our trainees and we want to make sure they assess patient safety first. We want the cases to be relevant and match teaching goals and objectives."

Zheng said he and the team of evaluators in the Department of Behavioral Medicine and Psychiatry think the AI tool will help medical residents learn more about the new role of chatbot in terms of a psychiatric conditions or clinical course trajectory.

"We feel like this can be incorporated in our didactics which is something to consider for the future because we want to make sure our physicians are AI competent," Zheng said. "This aligns very well with our goal to not only helping us to learn more about this emerging problem, but it also fits with the Accreditation Council for Graduate Medical Education goal with AI competency."

As more patients and psychiatrists turn to AI in mental healthcare, Hu and Zheng agree that more research is needed before it's standardized for mental health purposes.

"One of the next steps could be to collect and analyze patterns from cases reported in medical journals and news media," Hu said. "That information can help us guide AI to create better training cases. We also need to think about clinical variations, such as patient age, medical history, or existing medical conditions, because those factors can affect how we interpret the chatbot's role and develop coping strategies."

Zheng said he sees AI as something connected to our daily lives that requires great care.

"We are not asking people not to use AI, but recognizing the pathological use or potential harm is very important. For people with mental problems, I recommend that AI cannot replace professionals."

-WVU-

ls/8/4/26

MEDIA CONTACT: Linda Skidmore
Health Research Writer
WVU Health Sciences Center
[email protected]