The World's First! Tsinghua University/Shanghai Jiaotong University and Others Jointly Built a Visual-language Model for Diabetes Diagnosis and Treatment, Published in Nature
Diabetes is the fastest growing chronic disease in the world, which can cause blindness, renal failure, amputation, stroke, myocardial infarction, etc. It is also closely related to tumor infection. Among them, diabetic retinopathy (DR) is the most common progressive ocular microvascular complication in diabetic patients.Able to affect diabetic patients with 30-40%.
More importantly, the presence of DR also indicates an increased risk of other complications (such as kidney, heart, and brain), so regular DR screening has been recommended as a key part of primary diabetes care. However, due to the shortage of infrastructure and human resources, as well as high costs,DR screening is often neglected in low- and middle-income countries.
In recent years, artificial intelligence, especially deep learning, has played an increasingly important role in the management of diabetes and its complications. However, past solutions usually focus on the single field of diabetes complication screening or auxiliary management, and rarely integrate these two important aspects at the same time. How to effectively integrate the automatic generation of diabetes diagnosis and treatment opinions with the accurate diagnosis of diabetic eye complications, and then build a safe and controllable multimodal intelligent model to support primary care doctors in achieving one-stop auxiliary diagnosis and treatment services, is the current cutting-edge trend and important challenge in the international medical field.
In this context, the team led by Professor Huang Tianyin, Vice Provost and Director of the School of Medicine at Tsinghua University, the team led by Professor Sheng Bin, Department of Computer Science, School of Electrical Engineering, Shanghai Jiao Tong University/Key Laboratory of Artificial Intelligence of the Ministry of Education, the team led by Professor Jia Weiping and Professor Li Huating, the Sixth People's Hospital affiliated to Shanghai Jiao Tong University School of Medicine, and the team led by Professor Qin Yuzong, National University of Singapore and Singapore National Eye Centre, worked together toSuccessfully built the world's first visual-large language model integrated system DeepDR-LLM for diabetes diagnosis and treatment.
The relevant research results have been published in Nature Medicine under the title "Integrated image-based deep learning and language models for primary diabetes care".
The DeepDR-LLM system combines a large language model with deep learning technology based on fundus images to provide primary care physicians with personalized diabetes management advice and auxiliary diagnosis results for diabetic retinopathy.The system was retrospectively validated in a multicenter cohort covering seven countries in three major regions: Asia, Africa, and Europe.Through prospective real-world research verification in China's primary care scenarios, it provides high-quality evidence-based evidence for the application effect of multimodal large models in the vertical field of diabetes care for the first time. The DeepDR-LLM system is expected to significantly improve primary diabetes management and DR screening in low- and middle-income countries, and provide a revolutionary digital solution for global diabetes management in the future.
Research highlights:
* This study innovatively proposed the fusion adapter (Adaptor) and low-rank adaptation (Low-Rank Adaptation, LoRA) collaborative optimization technology
* The DeepDR-Transformer module introduces the Transformer model architecture and is trained on more than 500,000 fundus images to accurately detect fundus image quality, segment lesions, and perform DR grading diagnosis
* After incorporating the DeepDR-LLM system into the diabetes diagnosis and treatment process, it can significantly improve the self-management behavior of newly diagnosed diabetic patients and increase the referral compliance of DR patients
Paper address:
https://www.nature.com/articles/s41591-024-03139-8
Follow the official account and reply "Diabetes Diagnosis and Treatment LLM" to get the complete PDF
Dataset download address:
The open source project "awesome-ai4s" brings together more than 100 AI4S paper interpretations and provides massive data sets and tools:
https://github.com/hyperai/awesome-ai4s
Datasets: 14 independent cross-sectional datasets
This study included 14 independent cross-sectional data sets.It contains 7 standard fundus images of diabetic patients and 7 independent cross-sectional datasets of portable fundus images. For the standard fundus image datasets, 2 datasets were used to develop and internally validate the DeepDR-Transformer module: the Shanghai Integrated Model (SIM) cohort and the Shanghai Diabetes Prevention Program (SDPP) cohort.
also,The study also selected 12 multi-ethnic data sets for external validation.They are: the Nicaragua Diabetes Screening Project (NDSP) cohort, the Diabetic Retinopathy Progression Study (DRPS) cohort, the Wuhan Tongji Health Management (WTHM) cohort, the Peking Union Medical College Diabetes Management (PUDM) cohort, the China National Diabetes Complications Study (CNDCS) cohort, the Guangzhou Diabetic Eye Study (GDES) cohort, the Chinese University of Hong Kong-Sight-threatening Diabetic Retinopathy (CUHK-STD) cohort, the Singapore Eye Disease Epidemiology Study (SEED) cohort, the Singapore National Diabetic Retinopathy Screening Project (SiDRP) cohort, the Sankara Nesra Diabetic Retinopathy Epidemiology and Molecular Genetics Study (SN-DREAMS) cohort, the Thailand National Diabetic Retinopathy Screening Project (TNDRSP) cohort and the UK Biobank (UKB) cohort.
Six additional datasets were used for external validation:The Chinese Portable Diabetic Retinopathy Screening Study-Eastern (CPSSDRE) cohort, the Chinese Portable Diabetic Retinopathy Screening Study-Middle (CPSSDRM) cohort, the Chinese Portable Diabetic Retinopathy Screening Study-West (CPSSDRW) cohort, the Chinese Portable Diabetic Retinopathy Screening Study-Northeastern (CPSSDRN) cohort, the Algerian Diabetic Retinopathy Study (ADRS) cohort, and the Uzbekistan Diabetic Retinopathy Study (UDRS) cohort.
The CPSSDRE, CPSSDRM, CPSSDRW, and CPSSDRN cohorts were from a real-world DR screening project facilitated by Phoebusmed. For the ADRS and UDRS datasets, participants were recruited in regions of Algeria and Uzbekistan, respectively, and the fundus images were captured using a variety of desktop and handheld fundus cameras from Canon, Topcon, Carl Zeiss, Optomed, and MicroClear.
Model architecture: DeepDR-LLM consists of two modules: LLM and DeepDR-Transformer
The DeepDR-LLM system consists of two modules, as shown in the following figure:
* Module I (LLM module),Provide personalized management advice to patients with diabetes;
* Module II (DeepDR-Transformer module),Image quality assessment, lesion segmentation, and DR grading from standard or portable fundus images.
Supervised fine-tuning of the LLM module
First, the researchers developed the LLM module by fine-tuning LLaMA.
Module I is a LLM model enhanced with domain knowledge, which aims to formulate diabetes management recommendations based on various clinical metadata such as medical history, physical examination, laboratory tests, and DR and DME diagnosis results. Due to the lack of specific domain knowledge, the initial LLM (i.e., LLaMA) is not directly effective in generating diabetes management recommendations.
Given this gap,The researchers developed a supervised fine-tuning method to incorporate diabetes management-related knowledge into the training process of the LLM.This approach can enhance the model's ability to generate diabetes management recommendations by adding necessary domain knowledge to the base LLM. The dataset for supervised fine-tuning is 371,763 pairs of clinical data retrospectively obtained from 267,730 participants in Shanghai Sixth People's Hospital and Huadong Sanatorium, and management recommendations from the real world, which were de-identified after collection.
Since all parameters (i.e., the original weights of the LLM) are updated during LLM fine-tuning, this is obviously not optimal in terms of efficiency.The research team innovatively proposed the fusion adapter (Adaptor) and low-rank adaptation (LoRA) collaborative optimization technology.The DeepDR-LLM multimodal large model was constructed, which can adapt to large language models including LLaMA. The LLM module integrates the training network layer with the inherent weight parameters of the large language model, breaking through the bottleneck of multimodal large model optimization under the constraints of low computing power resources.
Development and training of the DeepDR-Transformer module
Module II can be used as a tool for analyzing fundus images in Module I for DR prediction, therefore,The researchers proposed a separate model called DeepDR-Transformer.The model can extract different features from fundus images after fine-tuning on a specific task.
The researchers used standard fundus images to train DeepDR-Transformer on four tasks: image quality assessment model (determining evaluability), DR grading prediction model, DME prediction model (determining whether it exists), and lesion segmentation model (microaneurysm, hemorrhage, cotton wool spots, and hard exudates). For each model, the researchers loaded the pre-trained weights from ImageNet and then performed end-to-end fine-tuning.
Integrate Modules I and II
The DeepDR-LLM system has two modes: integrated module I and module II.
In the integrated model with the participation of doctors,The outputs of module II (i.e., fundus image gradability; lesion segmentation of microaneurysms, cotton wool spots, hard exudates, and hemorrhages; DR grade; and DME grade) can assist physicians in generating DR/DME diagnostic results (i.e., fundus image gradability, DR grade, DME grade, and lesion presence or absence).
In automated integration mode,The DR/DME diagnosis results include fundus image grading, DR grading, DME grading divided by module II, and the presence of lesions segmented by module II. These DR/DME diagnosis results and other clinical metadata will be input into module I to generate personalized management recommendations for diabetic patients.
Research results: DeepDR-LLM system can improve grassroots DR screening capabilities and diabetes diagnosis and treatment levels
The research team invited Professor Juliana CN Chan from the Chinese University of Hong Kong, Professor Bao Yuqian from the Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Professor Jonathan E. Shaw from the Baker Institute of Heart and Diabetes in Australia, Professor Justin B. Echouffo-Tcheugui from Johns Hopkins University in the United States, and Professor Gavin Siew Wei Tan from the Singapore National Eye Center and other famous scholars in diabetes-related disciplines to form an international multidisciplinary expert committee.
The expert committee randomly selected 100 case samples from the Chinese Diabetes Chronic Complications Study cohort covering 31 provinces and regions in China, formed a diagnosis and treatment consensus for each case, and used this as the standard answer to conduct a blind scoring of the diagnosis and treatment opinions given by the DeepDR-LLM system and primary care physicians (PCPs).
First, the DeepDR-LLM system’s ability to provide diabetes management advice in both Chinese and English.The figure below summarizes the evaluation results of diabetes management recommendations generated by four different methods (DeepDR-LLM, LLaMA, PCP, and resident physician) in three different areas in English and Chinese: extent of inappropriate content, extent of missing content, and likelihood of possible harm.
In English,The DeepDR-LLM recommendations for 71% were considered to have no inappropriate content, which is higher than LLaMA (51%) but comparable to PCP (71%). In addition, the DeepDR-LLM recommendations for 36% were considered to have no missing content (PCP: 27%). Finally, the DeepDR-LLM recommendations for 57% were rated as "low likelihood" to cause harm, which is comparable to PCP's 55%.
In Chinese,77% of DeepDR-LLM recommendations were considered to have no inappropriate content, which is higher than LLaMA (66%) and PCP (54%). In addition, 63% of DeepDR-LLM recommendations were considered to have no missing content, compared to 46% for PCP. 88% of DeepDR-LLM recommendations were rated as "low likelihood" of causing harm, compared to 60% for PCP.
The figure below shows the total score (defined as the sum of the domain-specific scores) for management recommendations generated in 4 different ways.In English,Management recommendations made by DeepDR-LLM were significantly better than those made by LLaMA (P < 0.001) and comparable to those made by PCPs and endocrinology residents.
In Chinese,DeepDR-LLM made management recommendations significantly better than LLaMA (P < 0.001) and PCP (P = 0.010), but comparable to endocrinology residents.The quality of diagnosis and treatment opinions output by the DeepDR-LLM system reaches or exceeds that of primary care doctors.
Subsequently, the researchers conducted external tests on the DeepDR-LLM system using more than 500,000 fundus images from Chinese cities including Beijing, Shanghai, Guangzhou, Wuhan and Hong Kong, and six countries including Singapore, India, Thailand, the United Kingdom, Algeria and Uzbekistan.
To evaluate the effectiveness of DeepDR-Transformer as an adjunct tool for PCPs and specialized non-physician graders (who are currently used in many DR screening programs, such as the United Kingdom, Singapore, and Vietnam) in identifying referable DR.The researchers evaluated the accuracy and time efficiency of the grading process with and without the assistance of the DeepDR-Transformer module.As shown in the following figure:
The results showed that the sensitivity range of unassisted PCP was 37.2%-81.6%, and with the assistance of DeepDR-Transformer, the range was improved to 78.0%-98.4%. Similarly, the specificity was improved from the original 84.4%-94.8% (unassisted) to 90.4%-98.8% (assisted).
In addition, with the assistance of DeepDR-Transformer,The median time required for evaluation was reduced from 14.66 seconds to 11.31 seconds per eye——This shows that the system has significantly improved the accuracy and efficiency of DR grading, and its diagnostic ability can even reach the level of professional ophthalmologists.
In addition, the research team applied the integrated DeepDR-LLM system to real-world clinical processes and conducted a prospective study following 769 Chinese primary care diabetes patients. The results showed that after the DeepDR-LLM system was incorporated into the diabetes diagnosis and treatment process, it could significantly improve the self-management behavior of newly diagnosed diabetes patients and increase the referral compliance of DR patients.
Contributing Asian wisdom to the intelligent management of diabetes
Today, the increasing prevalence of diabetes poses a major challenge to public health in China and around the world. Artificial intelligence, especially deep learning, plays an increasingly important role in the management of diabetes and its complications. The experts from the Tsinghua University team, Shanghai Jiaotong University team, National University of Singapore and Singapore National Eye Centre team mentioned above have been working in this field for many years.
In 2017, Professor Huang Tianyin, then Medical Director of the Singapore National Eye Centre, and his teamThe company is the first in the world to realize automatic diagnosis of moderate to severe DR cases in multi-ethnic populations based on deep learning algorithms.The results were published in JAMA, which is a milestone in the history of smart medical development.
In 2018, the teams of Professors Jia Weiping and Li Huating carried out cross-medical and engineering collaborative innovation with the team of Professor Sheng Bin from the School of Electronic Information and Electrical Engineering of Shanghai Jiao Tong University, and joined hands with top international academic institutions such as the Singapore National Eye Centre. With the support of the Shanghai Science and Technology Commission and Shanghai Jiao Tong University, they were approved to establish the Shanghai "Belt and Road" International Joint Laboratory for Intelligent Prevention and Control of Metabolic-Related Diseases, which is committed to carrying out extensive cross-medical and engineering and international cooperation in the field of intelligent prevention and control of metabolic-related diseases.
Since its establishment, the joint laboratory has analyzed millions of data.Developed a transfer-enhanced multi-task deep learning system DeepDR,It can realize automatic diagnosis of the whole course of DR from mild to proliferative lesions, and provide real-time feedback on the quality of fundus images and identification and segmentation of fundus lesions. This technology is also applied to the International Diabetes Federation's "Global Diabetic Retinopathy Screening Project in Low- and Middle-Income Countries" and has been promoted to 48 countries.
This result was published in Nature Communications in 2021 under the title "A deep learning system for detecting diabetic retinopathy across the disease spectrum".
* Paper link:
https://www.nature.com/articles/s41467-021-23458-5
At the end of 2021, Professor Huang Tianyin was hired by Tsinghua University as a chair professor and leader of the medical discipline, and actively carried out translational research on the diagnosis and treatment of diabetes and eye disease complications enabled by artificial intelligence. Through continuous collaborative research, the Joint Laboratory and Professor Huang Tianyin's team at Tsinghua University successfully built a deep learning system DeepDR Plus based on the Weibull mixed distribution model, surpassing the previous leading technology in this field by the Google team in the United States. It is the first in the world to achieve risk warning and progression prediction of DR progression for up to 5 years, and can maintain an extremely low missed diagnosis rate while significantly reducing screening frequency and public health costs. The results were published in Nature Medicine in January 2024.
In short, the birth of DeepDR-LLM can be said to be the "culmination" of previous research results. Adhering to the concept of people-oriented and intelligent good, the researchers provide high-quality evidence-based basis for the future transformation of primary diabetes management, allowing global diabetes governance to better integrate into the trend of digitalization, intelligence, and greening, and contribute Asian wisdom to intelligent diabetes governance.
References:
1.https://mp.weixin.qq.com/s/MBtm0hY0gKE8NRQ8GfDy7A
2.https://www.nsfc.gov.cn/csc/20340/20343/68143/index.html
3.https://www.tsinghua.edu.cn/info/1182/112946.htm