VGL：整合组织病理学与基因表达的视觉-基因-语言多模态LLM用于肺癌细胞类型分类

Background Understanding the tumor microenvironment requires models that resolve cellular heterogeneity across molecular and spatial modalities. With the expansion of spatial transcriptomics, single-cell RNA-seq, and high-resolution histopathology imaging, there is a need for a unified foundation model that jointly interprets gene expression, spatial context, and visual tissue features. We developed a multimodal large language model (LLM) that integrates these modalities into a single adaptive framework handling heterogeneous inputs-including gene expression profiles, spatial transcriptomics spots, single-cell measurements, and histology patches-while generating harmonized outputs such as genes, cell types, and image-derived descriptors. Method We built a multimodal LLM within a Vision-Gene-Language (VGL) framework that integrates gene expression, histology images, and biological language representations. The model is based on MedGemma-4b-it and was fine-tuned using QLoRA for parameter-efficient training. Training used 5.2 million multimodal samples of H&E patches paired with highly variable gene expression profiles from non-small cell lung cancer, totaling 1,745,240 cells and spatial spots across scRNA-seq and spatial transcriptomics platforms (Visium, Xenium). The model was trained using multi-task learning across five canonical tasks spanning image-to-gene, gene-to-cell type, and cell type-to-gene objectives with task-specific ratio scheduling on 4 x H100 GPUs. We evaluated performance on cell type classification from gene expression profiles across 12 major immune and stromal cell types. Results The trained VGL model achieved 70.07% accuracy on the held-out test set (n=20,764) for predicting cell types from gene expression profiles, compared to 16.32% accuracy for the naive pre-trained model, a 4.3-fold improvement. Validation performance was similar (69.85% accuracy, n=41,529), indicating robust generalization. These gains demonstrate the value of a multimodal LLM that jointly leverages single-cell, spatial transcriptomics, and histology information. Through cross-modal learning and masking, the model learned gene and cell type embeddings that generalized across data platforms and spatial contexts and produced biologically consistent outputs even when only a subset of modalities was available. Conclusion We introduce a multimodal LLM-based spatial foundation model, VGL, that unifies single-cell RNA-seq, spatial transcriptomics, and histopathology imaging into a modality-agnostic framework. The improvement in cell type classification highlights the model's ability to capture and reason over cross-modal biological structure. This framework lays the groundwork for spatial AI systems that interpret heterogeneous molecular and imaging data and enable scalable tumor microenvironment profiling.