AI foundation model for single cell annotation from conventional histopathology images of cancer

Spatial imaging technologies have revolutionized our understanding of the tumor microenvironment (TME) by delineating its molecular and cellular architecture. Some of these approaches characterize tissue at single cell resolution. These platforms enable precise cell type annotation and, when spatially registered, allow molecular label transfer onto conventional histopathology images with hematoxylin and eosin (H&E) staining. However, spatial imaging approaches require complex instrumentation and have a high-cost per assay. These limitations prevent the application of spatial analyses on large sets of cancers. In contrast, conventional cancer histopathology slides are widely available and can be imaged at low cost. However, single cell identification from these images remains a manual, semi-quantitative process that is difficult to scale up.To address these limitations, we developed an AI foundation model that enables single-cell characterization directly from conventional H&E images. Our approach uses either spatial proteomic data (e.g., immunohistochemistry - IHC) or spatial transcriptomic profiling. We use these spatial assays to generate molecularly defined labels for diverse cell populations, including epithelial, lymphocyte, macrophage, and stromal cells. For this study, the molecular labels were the basis for training a spatial multimodal classifier. There are two tiers, (1) single-cell H&E crops were embedded using the foundation model H-optimous to obtain high-dimensional representations, and (2) a Multi-Layer Perceptron (MLP) neural network was trained on those embeddings for supervised cell type classification.We trained our model on a set of colorectal cancers (CRC), consisting of 40 multiplexed IHC slides and five Xenium spatial transcriptomic slides. Overall, we had 34 million single cells for model training. The model achieved an overall accuracy of 87.1% and a macro-average area under the receiver operating characteristic curve (AUROC) of 96.1%. Independent validation on seven CRC samples (~8 million cells) yielded consistent performance with 87.2% accuracy and 95% macro-average AUROC.In summary, our multimodal model enables automated and scalable single-cell-level cell type annotation directly from H&E images. This approach provides a quantitative foundation for immune-tumor interactions for colorectal cancer. Furthermore, by integrating H&E-derived cell type maps with tumor-specific genomic alterations from matched TCGA datasets, the framework enables systematic analysis of important TME cell types such as tumor-infiltrating lymphocytes and their spatial cellular distributions, offering insights into colorectal cancer microenvironmental architecture.