A deep learning-based multimodal integration framework for clinical outcome prediction

In oncology, a diverse range of advanced approaches, including medical imaging, genomic and transcriptomic profiling, and clinical data analysis, are utilized to comprehensively characterize each patient's tumor. This integrative strategy provides actionable insights that inform personalized care plans and therapeutic decisions, ultimately aiming to optimize patient outcomes. However, practical challenges persist, including how to integrate unstructured imaging data and address missing data modalities.Here, we present a new deep learning based, multimodal integration framework to address these challenges. Our approach incorporates a single modal loss, calculated between each modal representation and the patient's clinical outcome, which encourages the model to learn more clinical outcome relevant features from each modality. Furthermore, a mutual information estimator is implemented to enable the model to explore exclusive features within each data modality. For multimodal fusion, we leveraged a transformer architecture to combine different modalities' embeddings into unified patient-level representations. In our study, we specifically focused on whole slide images, gene expression and mutation. We utilized TCGA non-small cell lung cancer (NSCLC) data (n=989) to develop our model for predicting overall survival (OS), and validated its performance in three independent datasets: CPTAC (n=208), ConcertAI RWD360 ® linked Caris datasets (ConcertAI/Caris: n=2176), City of hope (COH, n=84). We observed high and robust performance across the four datasets (TCGA: 0.64 ± 0.03; CPTAC: 0.60 ± 0.04; ConcertAI/Caris: 0.59 ± 0.03; COH: 0.61 ± 0.04) using C-index as the evaluation metric. Further evaluating survival association in the three independent datasets indicated significant prognostic values of our model in both univariable (CPTAC: HR = 3.29, p = 0.007; ConcertAI/Caris: HR = 1.50, p = 5e-6; COH: HR = 3.16, p = 0.008) and multivariable (CPTAC: HR = 2.72, p = 0.03; ConcertAI/Caris: HR = 1.40, p = 0.002; COH: HR = 3.81, p = 0.02) Cox proportional hazards model after adjusting for known prognostic clinical factors.To interpret our model, we applied integrated gradients method to understand each feature's contribution to model output. Specifically, we identified 349 core OS-related genes in NSCLC, enriching in cancer-related pathways such as epithelial mesenchymal transition, focal adhesion and TNFA signaling via NFKB. Stratifying patient cohorts by treatment types allowed us to further identify exclusive genes per treatment. As a result, we identified 11 exclusive genes for immunotherapy, with enrichment in immune-related and metabolism pathways.In summary, we developed a multimodal integration framework that predicts clinical outcomes with high and robust performance. Interpretating our framework reveals potential prognostic and predictive markers to advance therapeutic development.