Ann-Christin Woerl

•A DLM was able to classify five of the most common STS subtypes from histology alone.•When aided by the DLM, pathologists were more accurate, faster, and more certain in their diagnosis.•A similar DLM was able to predict the disease-specific survival status in the most common STS subtype.•The DLM's prediction was an independent prognostic factor.•New image features associated with survival could be identified. BackgroundClinical management of soft tissue sarcoma (STS) is particularly challenging. Here, we used digital pathology and deep learning (DL) for diagnosis and prognosis prediction of STS.Patients and methodsOur retrospective, multicenter study included a total of 506 histopathological slides from 291 patients with STS. The Cancer Genome Atlas cohort (240 patients) served as training and validation set. A second, multicenter cohort (51 patients) served as an additional test set. The use of the DL model (DLM) as a clinical decision support system was evaluated by nine pathologists with different levels of expertise. For prognosis prediction, 139 slides from 85 patients with leiomyosarcoma (LMS) were used. Area under the receiver operating characteristic (AUROC) and accuracy served as main outcome measures.ResultsThe DLM achieved a mean AUROC of 0.97 (±0.01) and an accuracy of 79.9% (±6.1%) in diagnosing the five most common STS subtypes. The DLM significantly improved the accuracy of the pathologists from 46.3% (±15.5%) to 87.1% (±11.1%). Furthermore, they were significantly faster and more certain in their diagnosis. In LMS, the mean AUROC in predicting the disease-specific survival status was 0.91 (±0.1) and the accuracy was 88.9% (±9.9%). Cox regression showed the DLM’s prediction to be a significant independent prognostic factor (P = 0.008, hazard ratio 5.5, 95% confidence interval 1.56-19.7) in these patients, outperforming other risk factors.ConclusionsDL can be used to accurately diagnose frequent subtypes of STS from conventional histopathological slides. It might be used for prognosis prediction in LMS, the most prevalent STS subtype in our cohort. It can also help pathologists to make faster and more accurate diagnoses. This could substantially improve the clinical management of STS patients. Clinical management of soft tissue sarcoma (STS) is particularly challenging. Here, we used digital pathology and deep learning (DL) for diagnosis and prognosis prediction of STS. Our retrospective, multicenter study included a total of 506 histopathological slides from 291 patients with STS. The Cancer Genome Atlas cohort (240 patients) served as training and validation set. A second, multicenter cohort (51 patients) served as an additional test set. The use of the DL model (DLM) as a clinical decision support system was evaluated by nine pathologists with different levels of expertise. For prognosis prediction, 139 slides from 85 patients with leiomyosarcoma (LMS) were used. Area under the receiver operating characteristic (AUROC) and accuracy served as main outcome measures. The DLM achieved a mean AUROC of 0.97 (±0.01) and an accuracy of 79.9% (±6.1%) in diagnosing the five most common STS subtypes. The DLM significantly improved the accuracy of the pathologists from 46.3% (±15.5%) to 87.1% (±11.1%). Furthermore, they were significantly faster and more certain in their diagnosis. In LMS, the mean AUROC in predicting the disease-specific survival status was 0.91 (±0.1) and the accuracy was 88.9% (±9.9%). Cox regression showed the DLM’s prediction to be a significant independent prognostic factor (P = 0.008, hazard ratio 5.5, 95% confidence interval 1.56-19.7) in these patients, outperforming other risk factors. DL can be used to accurately diagnose frequent subtypes of STS from conventional histopathological slides. It might be used for prognosis prediction in LMS, the most prevalent STS subtype in our cohort. It can also help pathologists to make faster and more accurate diagnoses. This could substantially improve the clinical management of STS patients.

BACKGROUND: Clear-cell renal cell carcinoma (ccRCC) is common and associated with substantial mortality. TNM stage and histopathological grading have been the sole determinants of a patient's prognosis for decades and there are few prognostic biomarkers used in clinical routine. Management of ccRCC involves multiple disciplines such as urology, radiology, oncology, and pathology and each of these specialties generates highly complex medical data. Here, artificial intelligence (AI) could prove extremely powerful to extract meaningful information to benefit patients. OBJECTIVE: In the study, we developed and evaluated a multimodal deep learning model (MMDLM) for prognosis prediction in ccRCC. DESIGN SETTING AND PARTICIPANTS: Two mixed cohorts of non-metastatic and metastatic ccRCC patients were used: (1) The Cancer Genome Atlas cohort including 230 patients and (2) the Mainz cohort including 18 patients with ccRCC. For each of these patients, we trained the MMDLM on multiscale histopathological images, CT/MRI scans, and genomic data from whole exome sequencing. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS: Outcome measurements included Harrell's concordance index (C-index) and also various performance parameters for predicting the 5-year survival status (5YSS). Different visualization techniques were used to make our model more transparent. RESULTS: The MMDLM showed great performance in predicting the prognosis of ccRCC patients with a mean C-index of 0.7791 and a mean accuracy of 83.43%. Training on a combination of data from different sources yielded significantly better results compared to when only one source was used. Furthermore, the MMDLM's prediction was an independent prognostic factor outperforming other clinical parameters. INTERPRETATION: Multimodal deep learning can contribute to prognosis prediction in ccRCC and potentially help to improve the clinical management of this disease. PATIENT SUMMARY: An AI-based computer program can analyze various medical data (microscopic images, CT/MRI scans, and genomic data) simultaneously and thereby predict the survival time of patients with renal cancer.

In this paper, we examine gradients of logits of image classification CNNs by input pixel values. We observe that these fluctuate considerably with training randomness, such as the random initialization of the networks. We extend our study to gradients of intermediate layers, obtained via GradCAM, as well as popular network saliency estimators such as DeepLIFT, SHAP, LIME, Integrated Gradients, and SmoothGrad. While empirical noise levels vary, qualitatively different attributions to image features are still possible with all of these, which comes with implications for interpreting such attributions, in particular when seeking data-driven explanations of the phenomenon generating the data. Finally, we demonstrate that the observed artefacts can be removed by marginalization over the initialization distribution by simple stochastic integration.