Real-world evaluation of multimodal AI: Foundation model-driven multimodal AI for GBM, NSCLC, and PDAC

Purpose: Translating multimodal AI from curated research datasets to real-world clinical practice remains a critical challenge in precision oncology. In this study, we adapted HONeYBEE, a foundation model-driven multimodal AI platform, for real-world oncology workflows. We focused on three cancers, glioblastoma (GBM), non-small cell lung cancer (NSCLC), and pancreatic ductal adenocarcinoma (PDAC), using routine clinical documentation, radiology/ pathology reports, and imaging studies to improve survival prediction and cohort stratification. Methods: We curated 3 cohorts (GBM n=160, NSCLC n=580, PDAC n=171), spanning 911 patients from single NCI-designated Cancer Center. The framework processed multimodal embeddings generated via HONeYBEE. Unlike curated research datasets, these cohorts had incomplete available data (8.2-47% missing), heterogeneous documentation and imaging protocols. We employed cross-modal attention mechanisms to dynamically learn hierarchical relationships between modalities while incorporating 99.96% dimensionality reduction. Cross-validation was used to evaluate concordance index (C-index), risk stratification for survival outcomes, and three attribution methods that quantify per-modality contributions. Results: The framework achieved C-indices of 0.637±0.087 for GBM, 0.598±0.021 for NSCLC, and 0.679±0.029 for PDAC, demonstrating consistent performance across cancer types despite substantial missing data. Risk stratification for survival outcomes identified clinically meaningful groups with four-fold (GBM: low 28 months vs. high-risk 6 months), five-fold (NSCLC: low 60 months vs. high-risk 12 months), and three-fold (PDAC: low 100 months vs. high-risk 35 months) differences in median survival. Attribution analysis revealed disease-specific patterns reflecting clinical reality. Text reports dominated GBM predictions (43.7%), capturing critical clinical information, imaging data drove NSCLC predictions (49%), reflecting central role of CT in staging, and balanced contributions characterized PDAC (31-35% per modality), aligning with guidelines emphasizing comprehensive assessment. Patient-level attribution demonstrated that high-risk individuals relied heavily on adverse imaging features, while low-risk patients showed balanced modality contributions, providing actionable insights for clinical review. Conclusions: This work successfully extends research from datasets to real-world clinical environments, demonstrating practical utility for treatment stratification and prognostic assessment across three challenging malignancies. Framework's modular architecture enables seamless integration with existing systems. By generating standardized patient embeddings with incomplete and heterogeneous data, we provide a scalable infrastructure for deploying multimodal AI in routine oncology care.