Agent Psychological Clinic Is Online! Based on 1.3K Depression Consultation Dialogues, the Shanghai Jiaotong University Team Built a Large Model Dialogue Agent That Can Diagnose Depression
Mental health issues are one of the biggest challenges facing society today. According to the World Mental Health Report by the WHO, approximately 246 million people suffer from depressive disorders, with an average of 3,153 cases per 100,000 people. It can be said that this is one of the most common mental disorders.
Yet, mental health care resources remain inadequate today, especially in low- and middle-income countries.Professional mental health medical resources are mainly concentrated in cities and large institutions, resulting in an imbalance in the allocation of hospital resources.This makes it difficult for patients suffering from mental disorders such as depression to obtain appropriate medical assistance in a timely manner.
In order to better allocate medical resources and alleviate the dilemma of resource shortage, efficient and accurate automated depression diagnosis methods are important and critical. Among various automated diagnosis methods, conversational agents are considered to be very effective due to their cost-effectiveness, time saving and maintenance of user anonymity. It is worth noting that as the diagnostic conversation continues to deepen,The agent's dialogue strategy needs to change with the patient's mental state and the progress of the doctor-patient relationship.This undoubtedly poses a severe challenge to the construction of dialogue agents.
In response to the above problems,The team of Professor Mengyue Wu from X-LANCE Laboratory of Shanghai Jiao Tong University, together with the University of Texas at Arlington (UTA), Tianqiao Brain Science Institute (TCCI) and ThetaAI, built an automated large-model dialogue agent simulation system - Agent Mental Clinic (AMC).Used for preliminary diagnosis of depression. This system can simulate psychiatrists and potential depression patients at the same time, and simulate depression consultation dialogues. In addition, in order to optimize the dialogue process and guide psychiatrists, AMC has also designed a mentor role to provide guidance to psychiatrists for the next round of dialogue. All roles in the system can be played by real people or implemented by large models.
In order to better adapt to the needs of the diagnostic environment, AMC uses the ideas of popular large model agent construction, and also proposes a three-layer memory storage structure and a new memory retrieval mechanism to achieve more efficient and accurate depression consultation and initial screening. This dialogue system can not only be used for preliminary screening of patients with potential depression, but also for training intern psychiatrists or students, and provide some guidance and help before officially entering the department for internship. The role of instructor, which can be played by professional psychiatrists, also provides feasible optimization ideas for the psychiatrist agent based on the large model.
The related research, titled "Depression Diagnosis Dialogue Simulation: Self-improving Psychiatrist with Tertiary Memory", has been published as a preprint.
Research highlights:
* A new type of dialogue agent simulation system was built to simulate the diagnostic conversation between patient agent and psychiatrist agent, providing an effective new method for training trainee psychiatrists and preliminary screening of potential depression patients.
* Propose an innovative three-layer memory structure and memory retrieval module to enhance the agent's summarizing skills in the diagnosis stage, providing a new direction for the optimization of depression diagnosis and dialogue simulation in the future.
* The AMC system has improved both depression diagnosis and suicide prediction, and the framework can be applied to other specific domains and is suitable for training with limited labeled cases.
Paper address:
https://arxiv.org/abs/2409.15084
The open source project "awesome-ai4s" brings together more than 100 AI4S paper interpretations and provides massive data sets and tools:
https://github.com/hyperai/awesome-ai4s
D4 Data set, verified by professional doctors
D4 The data set was collected by Professor Wu Mengyue's team and quality-verified by professional doctors.The collection of the dialogue dataset has three stages:
* Distribute questionnaires on social media and other platforms to collect and construct personal portraits of real potential depression patients.
* Recruit and train simulated patients and simulated doctors, provide the simulated patients with the collected portraits of depression patients, and then have the simulated doctors and simulated patients conduct a depression consultation dialogue.
* The collected simulated conversations will be handed over to professional psychiatrists to verify the quality of the consultation conversations, and the severity of depression and symptom summary will be made for the conversations that meet the quality standards.
After collection and verification, Wu Mengyue's team collected a total of 1,339 conversations of satisfactory quality to form the depression consultation dataset D4. The degree of depression was divided into four categories: none (430 conversations), mild (342 conversations), moderate (368 conversations), and severe (199 conversations). The average number of conversation rounds was 21.6 rounds. Afterwards, Wu Mengyue's team proposed the SEO depression symptom diagnosis annotation framework to annotate the D4 dataset.
Innovative three-layer memory structure and memory retrieval module
The researchers proposed an automated large-model dialogue agent simulation system with a three-layer memory storage structure and a new memory retrieval mechanism, which includes three modules:
* Patient Agent:The patient agent was initialized using user information extracted from D4 user portraits and depression consultation dialogues. AMC selected 100 classic cases from D4 as a training set and used GPT-4 to extract event information from the consultation dialogue as the patient's initial memory node.
* Psychiatrist Agent:The description and diagnostic basis of depressive disorders in ICD-11 were used for initialization, and the patient's electronic medical records and summarized diagnostic skills were accumulated as the diagnosis progressed.
* Supervisor plugin:It is used to compare the diagnosis results of the psychiatrist Agent with the patient Agent, and summarize the experience and hand it over to the psychiatrist Agent for storage.
The operation process of the entire AMC system is shown in the figure, which has 6 steps in total:
1. Initialize a batch of patient agents using patient portraits generated by GPT-4 based on D4. The diagnosis results of depressive disorders will not be provided to the patient agent during the patient's consultation. 2. The patient agent and the psychiatrist agent have a depression consultation chat.
3. During the second step, the instructor plug-in will track the symptoms mentioned by the patient and provide the psychiatrist Agent with suggestions for the next step of the conversation based on the progress of the conversation.
4. After the consultation dialogue is over, the instructor plug-in compares the psychiatrist Agent's final diagnosis of the patient with the real doctor's diagnosis stored in D4. 5. Based on the comparison of the diagnosis results, the instructor plug-in summarizes the guidance provided to the psychiatrist Agent and provides it to the psychiatrist Agent.
6. After completing a consultation process, the psychiatrist calls the next patient Agent for diagnosis and repeats the process.
To better retrieve adapted memory to process conversations and diagnoses,The researchers constructed a three-layer agent memory architecture: conversation history, electronic medical records, and summary skills.in:
* Diagnosis History is the history record of the current dialogue;
* Electronic Medical Records (EMRs) are summaries of the conversations written by the psychiatrist agent after each patient consultation, including the patient's complaints, symptoms, and other event information;
* Diagnosis Skills are guidance opinions summarized by the instructor plug-in and stored in the psychiatrist agent's memory, which helps to optimize the conversation process and diagnosis accuracy.
Experimental conclusion: Improve the accuracy of diagnosis of depression and suicidal tendencies
To evaluate the effectiveness of AMC, the researchers conducted two sets of experiments on the D4 test set.
The first is to use the original dialogue of D4 as the consultation dialogue between the psychiatrist agent and the patient agent to diagnose, which is used to test the diagnostic ability of the model. The second is to diagnose through the dialogue between the psychiatrist agent and the patient agent, which comprehensively examines the role-playing ability and diagnostic ability of the agent.
The researchers calculated the diagnostic accuracy of depression and suicidal tendencies (categorized as none, mild, moderate, and severe). By comparing the AMC system with and without memory, the results showed that the overall performance of the model on the test set was improved.The effectiveness of AMC in utilizing the hierarchical memory architecture has been verified, and both simple diagnosis and dialogue + diagnosis can be improved to a certain extent.
In order to further verify the effectiveness of the three-layer memory structure,The researchers conducted an ablation experiment and found that adding the memory of electronic medical records and summary skills at the same time could bring a stable improvement in the accuracy of the model, proving the effectiveness of the three-layer memory structure.
To verify the effectiveness of the instructor plugin,The researchers also conducted plug-in ablation experiments. The experiments showed that the instructor plug-in also improved the architecture of the AMC system to a certain extent.
Discover the secrets of sound
In recent years, AI has demonstrated its powerful capabilities in the diagnosis and treatment of various diseases, such as the detection of respiratory and gastrointestinal diseases based on voice. However, the diagnosis and treatment of mental illness has been slow due to the lack of high-quality relevant data. The reason is that most patients with mental illness still have a sense of stigma, and the relevant diagnosis and treatment records involve patient privacy, so it is difficult to form a large-scale data set for AI model training.
The main research direction of Wu Mengyue's research group is computational psychiatry and pathological speech research in audio understanding.The D4 dataset used in the above-mentioned research is the world's first open source depression consultation dialogue dataset that meets clinical standards, built by her team, providing a solid foundation for related research.
It is worth mentioning that Teacher Wu Mengyue has rich personal experience, has a background in psychoacoustic research, and is committed to combining AI with mental health research. HyperAI conducted an in-depth interview with Professor Wu Mengyue.Click to view the full report: Wu Mengyue from Shanghai Jiao Tong University: Using speech intelligence technology to issue the first diagnosis and treatment order for mental illness.
The first author of this research paper is Lan Kunyao, a Ph.D. student in the Department of Computer Science, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University.He graduated from Shanghai Jiao Tong University with a bachelor's degree in information security. His main research direction is the dialogue system for diagnosis and treatment of mental illness. He won the second prize in the 2023 Mathematical Medicine Technology and Application Innovation Competition and the first prize in the 13th National College Student Information Security Competition.
The Shanghai Jiao Tong University X-LANCE Laboratory to which his research group belongs is known as the Shanghai Jiao Tong University Cross-Media Language Intelligence Laboratory.Founded in 2012, it was formerly the Intelligent Speech Laboratory (SpeechLab). After years of development, it has become a "cross-modal language intelligence laboratory" covering all core research areas of audio-visual language information processing.
At present, the faculty of the Cross-Media Language Intelligence Laboratory consists of 1 professor, 4 associate professors and 1 research assistant, with more than 20 doctoral students, more than 30 master students, and more than 30 undergraduates from ACM classes, AI classes, IEEE classes, CS majors of the School of Electrical Engineering, Paris Institute of Excellence, Michigan Institute, etc.
The laboratory has received support from many national and corporate projects, including the National Key R&D Program and the National Natural Science Foundation of China's Excellent Young Scientist Fund. The laboratory has in-depth cooperation with AIS Technology Co., Ltd. to establish the "Shanghai Jiaotong University AIS Intelligent Human-Computer Interaction Joint Laboratory". The laboratory can mobilize rich data resources and rich computing resources of up to hundreds of H800, A800, A10 and other GPU cards. It is one of the very few artificial intelligence laboratories in the world that can conduct industrial-level large-scale data analysis and research.
Students with a psychology background and an interest in computational mental health are welcome to join the research~
Laboratory homepage: