Omics-aware patch aggregation via multimodal co-training with a scalable multi-omics encoder for slide-level prediction across an oncology biomarker panel

Background: Whole-slide images (WSIs) are widely available, but matched multi-omics profiles are limited, especially when multiple modalities are required. We developed a multimodal learning framework that integrates RNA expression and DNA mutation with H&E WSIs, explicitly modeling partially observed omics to train an omics-aware patch aggregator for slide-level prediction. Methods: An omics-aware patch aggregator was trained with a slide-level contrastive objective that aligns a Multi-Omics Encoder (MOE) and a Slide Encoder (SE). The MOE is a shared Transformer that tokenizes each omics modality, concatenates modality-specific tokens with omic-type encodings, and uses self-attention to capture cross-omic interactions, allowing new omics to be added without changing the architecture. The SE processes tens of thousands of patches per slide and incorporates patch coordinates via relative positional bias to emphasize spatially proximal regions. Multimodal pretraining used ~20,000 partially paired WSIs and multi-omics profiles from The Cancer Genome Atlas and Genotype-Tissue Expression projects. Results: After multimodal pretraining, the omics-aware aggregator supported slide-level prediction across an oncology biomarker panel. For gene overexpression, area under the ROC curve (AUC) values were: LAG3 0.84, CLDN6 0.68, CD274 0.98, EGFR 0.74, ERBB2 0.72, ERBB3 0.69, CD276 0.82, VTCN1 0.72, TACSTD2 0.77, FOLR1 0.93, and MET 0.82. By tumor type and task, lung adenocarcinoma achieved AUCs of 0.70 for tumor mutational burden, 0.87 for EGFR mutation, and 0.62 for KRAS mutation; colorectal cancer 0.99 for microsatellite instability; breast cancer 0.95, 0.88, and 0.81 for ER, PR, and HER2 protein subtyping and 0.74 and 0.86 for TP53 and PIK3CA mutations; renal cell carcinoma 0.60 for PBRM1 and 0.74 for BAP1 mutations; and colon adenocarcinoma 0.88 for KRAS and 0.89 for TP53 mutations. Gene overexpression tasks used The Cancer Genome Atlas with five-fold cross-validation across four seeds; lung and colorectal models used Samsung Medical Center cohorts; breast subtyping used the BCNB cohort; and renal and colon tasks used Clinical Proteomic Tumor Analysis Consortium data. Conclusions: An omics-aware patch aggregation framework co-trained with a scalable multi-omics encoder and WSIs enables accurate slide-level prediction for diverse biomarkers and tumor types and illustrates how partially paired multi-omics data can strengthen digital pathology models. AI was used for language editing only; authors are responsible for all content and approved the final version.