Interpreting PLMs for cancer discovery: High attention hotspots predict pathogenic mutation positions and novel drug binding sites

Computational proteomics has revolutionized cancer research, guiding targeted experimental exploration to accelerate protein-based mechanistic discovery. Protein Language Models (PLMs) enable scalable, resource-efficient study; through large-scale training on only primary protein sequences, PLMs generate vector representations of protein structure that have been shown to capture biochemical and structural properties. A core component of PLMs is the attention mechanism, which specifically captures long-range interactions across a protein sequence in attention matrices. Using the previously unexplored attention matrices generated by the Evolutionary Scale Modelling 2 (ESM-2) PLM, we developed a novel method to identify High Attention (HA) residues, the specific residues that ESM-2 assigns the most attention to early in training. We found that HA residues had interpretable links to biological function across the human proteome, including proximity to active sites and conservation across protein families. We further used AlphaMissense pathogenicity predictions and TCGA-labeled pathogenic variant positions to determine that HA residues predict protein regions with high pathogenic risk. Finally, we explored the utility of HA residues for novel binding site discovery, an open challenge in cancer research. Using Uniprot and Biolip annotations, we confirmed that HA residues were spatially proximal to previously-known binding sites. We then used SiteMap to predict the bindability of HA residue regions in both annotated and unannotated proteins. We identified multiple cancer protein examples where HA residues identified regions with previously undiscovered high bindability and thus potential novel therapeutic utility. In summary, our work demonstrates the biological interpretability of PLM representations and offers a valuable method to prioritize functionally relevant protein residues for targeted biomedical research.