Artificial intelligence-based multi-modal multi-tasks analysis reveals tumor molecular heterogeneity, predicts preoperative lymph node metastasis and prognosis in papillary thyroid carcinoma: a retrospective studyYunfang Yu, Wenhao Ouyang, Yunxi Huang et al.|International Journal of Surgery|2024 BACKGROUND: Papillary thyroid carcinoma (PTC) is the predominant form of thyroid cancer globally, especially when lymph node metastasis (LNM) occurs. Molecular heterogeneity, driven by genetic alterations and tumor microenvironment components, contributes to the complexity of PTC. Understanding these complexities is essential for precise risk stratification and therapeutic decisions. METHODS: This study involved a comprehensive analysis of 521 patients with PTC from our hospital and 499 patients from The Cancer Genome Atlas (TCGA). The real-world cohort 1 comprised 256 patients with stage I-III PTC. Tissues from 252 patients were analyzed by DNA-based next-generation sequencing, and tissues from four patients were analyzed by single-cell RNA sequencing (scRNA-seq). Additionally, 586 PTC pathological sections were collected from TCGA, and 275 PTC pathological sections were collected from the real-world cohort 2. A deep-learning multi-modal model was developed using matched histopathology images, genomic, transcriptomic, and immune cell data to predict LNM and disease-free survival (DFS). RESULTS: This study included a total of 1011 PTC patients, comprising 256 patients from cohort 1, 275 patients from cohort 2, and 499 patients from TCGA. In cohort 1, the authors categorized PTC into four molecular subtypes based on BRAF, RAS, RET, and other mutations. BRAF mutations were significantly associated with LNM and impacted DFS. ScRNA-seq identified distinct T-cell subtypes and reduced B-cell diversity in BRAF-mutated PTC with LNM. The study also explored cancer-associated fibroblasts and macrophages, highlighting their associations with LNM. The deep-learning model was trained using 405 pathology slides and RNA sequences from 328 PTC patients and validated with 181 slides and RNA sequences from 140 PTC patients in the TCGA cohort. It achieved high accuracy, with an area under the curve (AUC) of 0.86 in the training cohort, 0.84 in the validation cohort, and 0.83 in the real-world cohort 2. High-risk patients in the training cohort had significantly lower DFS rates ( P <0.001). Model AUCs were 0.91 at 1 year, 0.93 at 3 years, and 0.87 at 5 years. In the validation cohort, high-risk patients also had lower DFS ( P <0.001); the AUCs were 0.89, 0.87, and 0.80 at 1, 3, and 5 years. The authors utilized the GradCAM algorithm to generate heatmaps from pathology-based deep-learning models, which visually highlighted high-risk tumor areas in PTC patients. This enhanced clinicians' understanding of the model's predictions and improved diagnostic accuracy, especially in cases with lymph node metastasis. CONCLUSION: The artificial intelligence (AI)-based analysis uncovered vital insights into PTC molecular heterogeneity, emphasizing BRAF mutations' impact. The integrated deep-learning model shows promise in predicting metastasis, offering valuable contributions to improved diagnostic and therapeutic strategies.
Unraveling the role of M1 macrophage and CXCL9 in predicting immune checkpoint inhibitor efficacy through multicohort analysis and single‐cell RNA sequencingThe exact function of M1 macrophages and CXCL9 in forecasting the effectiveness of immune checkpoint inhibitors (ICIs) is still not thoroughly investigated. We investigated the potential of M1 macrophage and C-X-C Motif Chemokine Ligand 9 (CXCL9) as predictive markers for ICI efficacy, employing a comprehensive approach integrating multicohort analysis and single-cell RNA sequencing. A significant correlation between high M1 macrophage and improved overall survival (OS) and objective response rate (ORR) was found. M1 macrophage expression was most pronounced in the immune-inflamed phenotype, aligning with increased expression of immune checkpoints. Furthermore, CXCL9 was identified as a key marker gene that positively correlated with M1 macrophage and response to ICIs, while also exhibiting associations with immune-related pathways and immune cell infiltration. Additionally, through exploring RNA epigenetic modifications, we identified Apolipoprotein B MRNA Editing Enzyme Catalytic Subunit 3G (APOBEC3G) as linked to ICI response, with high expression correlating with improved OS and immune-related pathways. Moreover, a novel model based on M1 macrophage, CXCL9, and APOBEC3G-related genes was developed using multi-level attention graph neural network, which showed promising predictive ability for ORR. This study illuminates the pivotal contributions of M1 macrophages and CXCL9 in shaping an immune-active microenvironment, correlating with enhanced ICI efficacy. The combination of M1 macrophage, CXCL9, and APOBEC3G provides a novel model for predicting clinical outcomes of ICI therapy, facilitating personalized immunotherapy.
Deep learning-based multi-modal data integration enhancing breast cancer disease-free survival predictionZehua Wang, Ruichong Lin, Yanchun Li et al.|Precision Clinical Medicine|2024 Abstract Background The prognosis of breast cancer is often unfavorable, emphasizing the need for early metastasis risk detection and accurate treatment predictions. This study aimed to develop a novel multi-modal deep learning model using preoperative data to predict disease-free survival (DFS). Methods We retrospectively collected pathology imaging, molecular and clinical data from The Cancer Genome Atlas and one independent institution in China. We developed a novel Deep Learning Clinical Medicine Based Pathological Gene Multi-modal (DeepClinMed-PGM) model for DFS prediction, integrating clinicopathological data with molecular insights. The patients included the training cohort (n = 741), internal validation cohort (n = 184), and external testing cohort (n = 95). Result Integrating multi-modal data into the DeepClinMed-PGM model significantly improved area under the receiver operating characteristic curve (AUC) values. In the training cohort, AUC values for 1-, 3-, and 5-year DFS predictions increased to 0.979, 0.957, and 0.871, while in the external testing cohort, the values reached 0.851, 0.878, and 0.938 for 1-, 2-, and 3-year DFS predictions, respectively. The DeepClinMed-PGM's robust discriminative capabilities were consistently evident across various cohorts, including the training cohort [hazard ratio (HR) 0.027, 95% confidence interval (CI) 0.0016–0.046, P &lt; 0.0001], the internal validation cohort (HR 0.117, 95% CI 0.041–0.334, P &lt; 0.0001), and the external cohort (HR 0.061, 95% CI 0.017–0.218, P &lt; 0.0001). Additionally, the DeepClinMed-PGM model demonstrated C-index values of 0.925, 0.823, and 0.864 within the three cohorts, respectively. Conclusion This study introduces an approach to breast cancer prognosis, integrating imaging and molecular and clinical data for enhanced predictive accuracy, offering promise for personalized treatment strategies.
MRI-based multimodal AI model enables prediction of recurrence risk and adjuvant therapy in breast cancerYunfang Yu, Wei Ren, Luhui Mao et al.|Pharmacological Research|2025 Timely intervention and improved prognosis for breast cancer patients rely on early metastasis risk detection and accurate treatment predictions. This study introduces an advanced multimodal MRI and AI-driven 3D deep learning model, termed the 3D-MMR-model, designed to predict recurrence risk in non-metastatic breast cancer patients. We conducted a multicenter study involving 1,199 non-metastatic breast cancer patients from four institutions in China, with comprehensive MRI and clinical data retrospectively collected. Our model employed multimodal-data fusion, utilizing contrast-enhanced T1-weighted imaging (T1+C) and T2-weighted imaging (T2WI) volumes, processed through a modified 3D-UNet for tumor segmentation and a DenseNet121-based architecture for disease-free survival (DFS) prediction. Additionally, we performed RNA-seq analysis to delve further into the relationship between concentrated hotspots within the tumor region and the tumor microenvironment. The 3D-MR-model demonstrated superior predictive performance, with time-dependent ROC analysis yielding AUC values of 0.90, 0.89, and 0.88 for 2-, 3-, and 4-year DFS predictions, respectively, in the training cohort. External validation cohorts corroborated these findings, highlighting the model’s robustness across diverse clinical settings. Integration of clinicopathological features further enhanced the model’s accuracy, with a multimodal approach significantly improving risk stratification and decision-making in clinical practice. Visualization techniques provided insights into the decision-making process, correlating predictions with tumor microenvironment characteristics. In summary, the 3D-MMR-model represents a significant advancement in breast cancer prognosis, combining cutting-edge AI technology with multimodal imaging to deliver precise and clinically relevant predictions of recurrence risk. This innovative approach holds promise for enhancing patient outcomes and guiding individualized treatment plans in breast cancer care. • A multi-task 3D deep learning MRI-based multimodal data model (3D-MMR-model) was developed for tumor segmentation and disease-free survival (DFS) prediction. An innovative multimodal approach integrating clinicopathological data with deep learning MRI insights yielded the 3D-MMR-model, surpassing standalone MRI models in predictive precision. Visualization techniques provided insight into decision-making processes, correlating model predictions with the tumor microenvironment.
Multimodal data fusion AI model uncovers tumor microenvironment immunotyping heterogeneity and enhanced risk stratification of breast cancerBreast cancer is the leading cancer among women, with a significant number experiencing recurrence and metastasis, thereby reducing survival rates. This study focuses on the role of long noncoding RNAs (lncRNAs) in breast cancer immunotherapy response. We conducted an analysis involving 1027 patients from Sun Yat-sen Memorial Hospital, Sun Yat-sen University, and The Cancer Genome Atlas, utilizing RNA sequencing and pathology whole-slide images. We employed unsupervised clustering to identify distinct lncRNA expression patterns and developed an AI-based pathology model using convolutional neural networks to predict immune-metabolic subtypes. Additionally, we created a multimodal model integrating lncRNA data, immune-cell scores, clinical information, and pathology images for prognostic prediction. Our findings revealed four unique immune-metabolic subtypes, and the AI model demonstrated high predictive accuracy, highlighting the significant impact of lncRNAs on antitumor immunity and metabolic states within the tumor microenvironment. The AI-based pathology model, DeepClinMed-IM, exhibited high accuracy in predicting these subtypes. Additionally, the multimodal model, DeepClinMed-PGM, integrating pathology images, lncRNA data, immune-cell scores, and clinical information, showed superior prognostic performance. In conclusion, these AI models provide a robust foundation for precise prognostication and the identification of potential candidates for immunotherapy, advancing breast cancer research and treatment strategies.