Multimodal AI predicts immune checkpoint inhibitor response from clinically available inputs and whole-slide images with explainable tumor biology and combination therapy insights

Background: Immune checkpoint inhibitors (ICIs) provide durable benefit for a subset of patients, yet most do not respond and current biomarkers (PD-L1, TMB, MSI) have limited predictive value. We developed AIM-io, a multimodal, biologically interpretable AI model that predicts ICI response using routine clinical inputs including liquid biopsy, tissue NGS and, for proof-of-concept, whole-slide images (WSI) processed with publicly available research-only foundation-model embeddings. AIM-io integrates reconstructed gene-expression and TME programs with predicted small-molecule sensitivities, enabling biological interpretation of each prediction. Methods: A pan-cancer real-world cohort (~3,000 patients, 13 tumor types) treated with anti-PD-(L)1 or anti-CTLA-4 therapy was assembled with linked rwOS/rwPFS. Inputs included liquid-biopsy, tissue NGS, or WSIs encoded embeddings (e.g., GigaPath, UNI). AIM-io incorporated clinical variables, genomic alterations from commercial LDTs, reconstructed expression/TME signatures (AIM-Ex), ICI drug/target embeddings, and AIM-Bx-derived small-molecule sensitivity profiles (~100 agents). Performance was assessed using C-index, hazard ratios, KM stratification, and comparisons to available biomarkers. Results: AIM-io showed robust cross-modality performance. In held-out LDT data, AIM-io significantly separated responders vs. non-responders (p < 10 -7 , HR = 0.23, 95% CI 0.14-0.41), outperforming PD-L1 (p = 0.26, HR = 0.77, 95% CI 0.48-1.22) and TMB (p < 10 -3 , HR = 0.39, 95% CI 0.24-0.65). Using liquid-biopsy DNA, AIM-io significantly stratified outcomes (p < 0.05, HR = 0.63, 95% CI 0.39-1.01). Results were consistent across cancer types and NGS platforms. With WSI-only inputs, AIM-io achieved significant outcome separation (HR = 0.46, p < 0.05; 95% CI 0.22-0.97) and outperformed models using WSI embeddings alone (HR = 0.62, p = 0.12), demonstrating feasibility in settings lacking molecular assays. Predicted responders showed enrichment for reconstructed TME programs (e.g., lymphocyte infiltration), whereas non-responders showed immunosuppressive signatures (e.g., TGF-beta, wound healing). Small-molecule sensitivity predictions differed, with responders showing greater predicted PARPi sensitivity and non-responders enriched for VEGFi sensitivity. Conclusions: AIM-io provides an explainable multimodal framework for ICI-response prediction using diverse, clinically accessible inputs-including liquid biopsy DNA and WSI embeddings. By integrating reconstructed expression, TME biology, and predicted therapeutic vulnerabilities, AIM-io offers an assay-agnostic approach for retrospective evaluation of immunotherapy response and hypothesis generation for rational combinations. Prospective validation is warranted.