AI-assisted pathology report abstraction for breast cancer

Background: Clinical cancer research often involves time-consuming manual abstraction of tumor information from pathology reports. Large language model (LLM)-based abstraction offers an efficient alternative but lacks transparency due to the “black-box” nature of LLMs. We evaluated the speed and accuracy of a combined approach using BRIM, an LLM-assisted human-in-the-loop clinical abstraction tool. Methods: We transcribed text from 828 PDF breast tumor pathology reports (399 previously abstracted by a trained tumor registrar) for 588 participants in the Carolina Breast Cancer Study, Phase 4 (CBCS4). We designed prompts to abstract tumor size, grade, estrogen and progesterone receptor (ER, PR) and human epidermal growth factor receptor 2 (HER2) information from reports, including percent positivity, staining intensity, immunohistochemistry (IHC) scores, as appropriate. Prompts were tuned through iterative testing in BRIM in conjunction with GPT-OSS-20B (an open-source LLM), where abstractors reviewed in-text evidence and model reasoning, providing real-time feedback to improve subsequent outputs. Performance metrics included abstraction speed and accuracy compared to gold-standard manual abstraction . Only cases with available data from both the LLM and registrar were included. Results: LLM-assisted abstraction of the 828 notes took 11.5 hours compared to 100 hours by our certified tumor registrar (average 7 min/report), representing a 90% reduction in abstraction time. As shown in Table, abstraction accuracies ranged from 85% (% positivity for PR) to 98% (tumor grade, ER intensity, and ER categorical interpretation), with most accuracies exceeding 90%. Conclusion: LLM-assisted abstraction of tumor variables from pathology reports is feasible, accurate, and may substantially reduce the burden of manual abstraction. Further work incorporating additional complexity (e.g. addenda, multifocality) will be important for more closely mimicking registrar workflows. Accuracy of LLM-based Abstraction Scope Accuracy (Correct Label/All Generated Labels) ER Status % Positive Per Note 89% (184/207) Intensity Per Note 98% (145/148) Category/Text Per Note 98% (196/200) PR Status % Positive Per Note 85% (168/198) Intensity Per Note 92% (126/137) Category/Text Per Note 89% (170/191) HER2 Status Intensity Per Note 97% (148/153) ISH Per Note 97% (30/31) HER2/CEP Ratio Per Note 94% (31/33) Tumor Grade Per Note 98% (336/344) Size (Numeric) Per Note 94% (263/279) ER Status Summary Per Patient 97% (308/316) PR Status Summary Per Patient 95% (299/313)