A practical guideline for benchmarking unimodal and multimodal foundation models on spatial transcriptomics data in oncology

Understanding the molecular and morphological structure of the tumor microenvironment is increasingly approached by the use of foundation models (FMs): large-scale, complex machine learning architectures trained on H&E, spatial omics, and/or molecular data. In parallel, data generation and experimental assays measuring molecular features in their spatial context are growing in size, resolution and scale, often no longer limited to only a few hundred genes, but multiple thousand features measured at subcellular resolution. This combination of scaled data generation and complex, often black-box, models requires well-defined benchmarks and transparent baselines to guide practitioners in deciding which approach(es) will best suit a specific goal with specific data. However, existing benchmarks are designed in a computational driven manner and focus only on a coarse-grained set of labels and genes. Although such benchmarks can give a good indication on overall performance, they often fail to capture different disease types and downstream use-cases which in practice will require distinct data types, outputs, and evaluation metrics. They also fail to capture the higher-level aim of foundation models: enabling novel biological findings. Finally, reproducing benchmark results on internal or novel datasets is often limited by complex computational frameworks. This work provides recommendations for building meaningful benchmarks for foundation models on spatial transcriptomics data, as defined by the combination of (a) meaningful, simple baselines, (b) diverse datasets with respect to indication and data type, and (c) meaningful downstream tasks and interpretable metrics. We begin by providing a complete overview of current benchmarking efforts leveraging spatial transcriptomics data and their design choices as well as limitations. To enhance existing benchmarks, we introduce a new set of tasks and metrics designed on common cell and gene signatures present in oncology, and allow for different processing choices designed for subcellular spatial transcriptomics data. For each task we also define a simple, memory-efficient baseline in order to put the performance of current and upcoming foundation models into perspective. We conclude by discussing useful guidelines for defining future benchmarking datasets for foundation models. The novel tasks and benchmarking frameworks introduced in this work provide a crucial toolset to researchers seeking to accurately assess the evolving landscape of uni- and multimodal foundation models for spatial omics.