Helping With Initial Screening for Depression! A Team From Shanghai Jiao Tong University Built an Agent Psychological Clinic. The First Author of the Paper Demonstrated the Demo Online and Shared Technical Highlights.
"I hit her, and even choked her neck and pushed her into a corner several times. Every time I did that, I would regret it. Why did I hit her? Why couldn't I control myself? Am I a bastard, a lunatic, an unforgivable person? But I really don't know what to do."This is what 18-year-old Haoran said when facing the CCTV camera.
In August this year, CCTV News "Relativity" reporter Zhuang Shengchun conducted an in-depth investigation into the real situation of teenage depression. Haoran was one of the few teenage depression patients who was willing to face the camera and tell his story.
Nowadays, depression is gradually becoming younger. According to the "China Youth Development Report", about 30 million children and adolescents under the age of 17 are suffering from various emotional disorders and behavioral problems. Adolescents are at a turbulent age and are relatively fragile. If the problem is not identified and intervened in time, it will further develop into a mental illness. However, there is a clear shortage of mental health medical resources at present, and most of the resources are mainly concentrated in cities and large medical institutions. This uneven distribution makes it difficult for many patients with depression to obtain medical assistance in a timely manner.
In response to this, Professor Wu Mengyue's team from the X-LANCE Laboratory at Shanghai Jiao Tong University collaborated with the University of Texas at Arlington (UTA), Tianqiao Brain Science Institute (TCCI) and ThetaAI.An automated large-model dialogue agent simulation system, the Agent Mental Clinic (AMC), was built for the preliminary diagnosis of depression.
In the fourth Meet AI4S live broadcast, HyperAI invited the first author of this research paper, Lan Kunyao, a doctor from the Cross-Media Language Intelligence Laboratory of Shanghai Jiao Tong University,With the theme of "Mental Health Diagnosis and Consultation Platform Based on Big Model Agent", he introduced in detail the platform's usage steps, technical highlights, and future plans.
Participate in a simulated consultation in Agent Psychology Clinic based on role-playing
In the past, we often used self-assessment forms (such as PHQ9, HAM-D) to determine whether we needed further medical resources, such as mental health tests when enrolling or joining a job. You will find that in order to prevent random filling of forms, many questions in the self-assessment form will be asked repeatedly to ensure consistency. This will lead to some problems - the workload is doubled and users find it boring.
Now, with the help of human-computer interaction, that is, chatting with a dialogue robot to simulate the consultation process,Doctor-patient communication has become more interesting, and with the development of large language models, the conversation experience is constantly improving.
The Intelligent Psychological Clinic (AMC) proposed by Professor Wu Mengyue's team can be used for the preliminary diagnosis of depression. Its main form is to simulate a theater performance and perform tasks similar to role-playing. During this process, users can talk to the virtual actor NPC to promote the consultation process. Specifically,The researchers mainly set up three "roles": patient agent, psychiatrist agent, and instructor agent.
Users can choose one of these three roles to experience.
The detailed information of these three agents is as follows:
* The patient Agent has some emotional confusion.You need to find a doctor to confirm whether you have a disease and determine whether you need further treatment. The researchers gave it different user portraits, allowing it to simulate multiple different patients who may face different problems and may be in different age groups.
Specifically, the researchers used a technique called D4 The dataset,That is, a group of potential patients are recruited through social media or public platforms, covering information such as gender, occupation, main demands and age. These portraits are strictly reviewed and verified before being used to simulate patients. Recruit and train simulated patients and simulated doctors, and both parties conduct simulated consultation dialogues. Professional psychologists evaluate the quality of these dialogues to ensure that they conform to the real doctor-patient communication scenarios. If they do, professional doctors will further determine the patient's depression and self-harm tendencies. These diagnoses are divided into 4 categories: barrier-free, mild, moderate, and severe.
*D4 It is the world's first open source depression consultation dialogue dataset that meets clinical standards.
Dataset address:https://x-lance.github.io/D4/
When initializing the patient agent, the researchers will provide the patient agent with the above-mentioned interview results, but will not tell the specific values of depression and self-harm tendencies to avoid data leakage. In addition, the patient agent's symptoms are usually manifested in diet, sleep, mood, interests, physical condition, etc., but this information is not enough to fully describe the patient. They also face pressure in life, work, and study, and these factors are usually reflected in the conversation rather than the specific symptoms.
To this end, the researchers chose to use GPT-4 to analyze the content of the conversation and find out the memory fragments related to the event, and integrate these memories into the patient agent's memory module, such as information about marital betrayal, setbacks in work and study, etc.
* The setting of the psychiatrist Agent is that of a new doctor with no consultation experience.The purpose is to test whether the doctor can make significant progress in communicating with patients.
On the doctor side, the Agent will not be given much practical experience in the initial stage, but researchers will provide some basic professional information, such as text descriptions of single-episode depressive disorder and recurrent depressive disorder, as part of the initial memory of the doctor Agent. As the number of interactions with patients increases, the skills and experience of the doctor Agent will continue to accumulate, thereby improving the level of consultation.
* The main purpose of the instructor agent is to control the flow of the doctor-patient dialogue.Prevent the conversation from going on indefinitely. In the future, if it is expanded to family counseling scenarios, such as when parents and children or couples consult together, the interaction between multiple agents may cause conflicts, and the instructor agent can help coordinate the conversation process.
During the consultation process,The doctor first asks questions, the patient responds, and the instructor then tracks the patient's answers and records his or her disease symptoms. After comparing the gap between the patient's symptoms and standard tests such as ICD-11 and DSM-5, the instructor will point out which issues have not been sorted out and give the doctor guidance on the next question. Based on these instructions, the doctor asks the patient new questions, forming an internal cycle. At the end of each conversation, the doctor will assess the patient's depressive and self-harm tendencies.
The instructor receives these assessment results and compares them with the D4 The instructor will compare the real doctor's diagnosis results stored in the data set. Based on the differences between the conversation history and the diagnosis results, the instructor will evaluate the doctor's performance during the consultation process, including language expression, questioning methods, and diagnostic accuracy. Finally, the reflected memory will be stored in the doctor's diagnostic skills to help the doctor agent gain more experience and growth during the consultation process.
It is worth mentioning that in order to better retrieve adaptive memory to process conversations and diagnoses, the researchers also proposed an innovative three-layer memory structure and memory retrieval module, namely conversation history, electronic medical records, and summary skills.
The platform continues to be optimized, and the future is promising
During the role-playing process, the researchers also encountered several problems and will optimize them in the future.
First, there is the problem of hallucinations.That is, the model will produce replies that are inconsistent with the facts during the conversation, which is a common problem for all large models. For example, in the past month, the patient may have thoughts of self-harm but have not carried them out, but the model may still incorrectly reply that the patient has self-harm behaviors every month. This false alarm is more obvious in long conversations. In order to deal with the phenomenon of hallucinations, when encountering similar problems, the researchers use some key symptoms in the user portrait as prompts to further prompt the model, but this approach can only temporarily alleviate the phenomenon of hallucinations, and further efforts are still needed in the future.
Another issue is the language style.Patients often do not use technical terms to describe their condition when they first visit the doctor, but the model tends to use technical terms such as "neuromotor retardation", which does not match the patient's performance during the first consultation.
In addition, the researchers observed that the model had a tendency to ask repeated questions.For example, influenced by past historical conversations, the model's opening and closing remarks are relatively fixed, and the way of asking questions is relatively consistent. However, in real conversations, these elements may change, and the responses are also diverse. How to provide appropriate comfort to make the conversation less boring is also a problem that needs to be considered in the future.
The main problems faced in optimizing the above content are inconsistent evaluation standards and scarcity of professional scenario data sets.In May last year, Dr. Chen Siyuan from Professor Wu Mengyue's team simulated doctor-patient conversations based on a large language model. The research results found that the large model would have obvious differences in terms from human terms. By modifying the prompt and prompting the model to use colloquial expressions, this symptom can be partially alleviated.
Regarding the issue of inconsistent evaluation standards, people usually have a general understanding of "high-quality dialogue", but there are significant differences in the evaluation standards for "medium" and "poor" dialogues. This may be a subjective error in individual evaluations, so larger-scale human testing is needed. In addition, many studies use high-quality models such as GPT-4 to evaluate other models, but large models are relatively weak in interpretability as black boxes, and the quality and number of human evaluators are limited. Therefore, there is still a lack of a large-scale automated evaluation standard.
In addition, the data sets required for different application scenarios vary, especially in professional scenarios, where the available sample size is often small, which may affect the reliability of the test and also limit the training of large models. In this regard, Lan Kunyao believes that a future solution is to introduce the concept of psychological cognitive models into role-playing practice. If a more general cognitive model can be established to adapt to different scenarios and effectively map existing network resources to this model, then the acquisition of subsequent training and test sets will become easier, which is a direction worth exploring.
“For me, what I’ve always wanted to explore further is whether these probabilistic models can eventually form a complete, independent personality thinking, and I hope to find a more stable way to evaluate the personality characteristics inherent in large language models.”Lan Kunyao said that in his opinion, since these models have learned a large amount of user corpus and user data, they should theoretically be able to show a personality trait. Based on this, what needs to be discussed is how to evaluate whether the model has this personality trait and how this personality trait affects its performance in answering questions, cognitive processes, and responding to user needs (such as psychological comfort). "If we can locate and distinguish the personality traits or personality modules of the model, we can adapt it to different scenarios and needs and perform different tasks, which is more effective than relying solely on prompt tuning."
Changing the world with technology
"I studied image processing in my undergraduate studies, and gradually became more interested in natural language processing during my graduate studies. My research direction at the time was towards smart education, and during that time I encountered a topic related to students' personalized learning needs. Later, when I was designing a learning companion for students,I found that students not only need academic help, but also psychological support.So I joined Professor Wu Mengyue's team during my doctoral studies, hoping to conduct more in-depth exploration in the field of psychology." Dr. Lan Kunyao said this when talking about the original intention of building the large-scale Agent Psychology Clinic.
Professor Wu Mengyue is a rare professional with an interdisciplinary background in psychology and computer science. In the process of promoting AI-enabled diagnosis and treatment of mental illness, she can accurately address the actual needs of patients and flexibly adjust research strategies.
The Shanghai Jiao Tong University Cross Media Language Intelligence Lab (SJTU Cross Media Language Intelligence Lab, X-LANCE) where she works has now become the "Cross-modal Language Intelligence Lab", covering the core research areas of audio-visual language information processing, and also involving the field of material chemistry. The team is committed to doing top-level technology research that can change the world. The laboratory's academic creed: To change the world with technology, one must first be an excellent engineer, and an outstanding engineer must be a scientist.
Laboratory homepage:https://x-lance.sjtu.edu.cn
The laboratory has received support from many national and corporate projects, including the National Key R&D Program and the National Natural Science Foundation of China's Excellent Young Scientist Fund. It has also cooperated deeply with AIS Technology Co., Ltd. to establish the "Shanghai Jiao Tong University AIS Intelligent Human-Computer Interaction Joint Laboratory". It has rich data resources and a large amount of computing resources, including hundreds of H800, A800 and A10 GPU cards. It is one of the very few artificial intelligence laboratories in the world that can conduct industrial-level large-scale data analysis and research.