Integrating image and text-based AI improves identification of metastatic sites from whole-slide pathology images

Metastatic cancer accounts for most cancer-related mortality, and determining the most likely metastatic destination of a tumor is essential for prognosis and treatment planning. Although every tumor arises from a primary site, its biological behavior varies widely: some remain localized, whereas others spread to distant organs following well-recognized patterns of organ-specific tropism. In clinical practice, distinguishing these behaviors relies on whole-slide images (WSIs), immunohistochemistry, and clinical information. However, metastatic lesions often lack distinctive histological features, making it difficult to determine whether a tumor has metastasized and, if so, its site of spread-especially in poorly differentiated biopsies. We hypothesize that tumors retain subtle morphological cues-rooted in metastatic potential and organ-specific tropism-that can help distinguish localized from metastatic phenotypes and, when metastasis is present, indicate the likely site of dissemination. Leveraging computational pathology and AI-based models, these latent features can be uncovered to improve tumor stratification and metastatic site prediction. We propose a novel image-and-text-based AI model that analyzes each region within a WSI to determine metastatic status and site. Our model utilizes medically meaningful textual descriptions-textual prototypes-generated by a pre-trained pathology AI model. In our study of 3,804 metastatic cases across six clinically relevant sites, each WSI was converted into patch-level image features using a pre-trained pathology AI model. After identifying metastatic disease, the model compares each patch with concise text descriptions of metastatic patterns (e.g., lymph node metastasis). A visual-textual similarity matrix quantifies how closely each patch matches these descriptions, guiding attention toward the regions most indicative of the metastatic site. Our model achieves an AUC of 88%, an accuracy of 74%, and a macro-F1 of 60%. These findings demonstrate improved metastatic-site classification by providing metastatic-specific semantic cues that direct attention to diagnostically important regions. We believe our model can suggest the most likely metastatic site using only routine pathology slides and offers a practical, scalable strategy for AI-assisted diagnosis.