An agentic AI workflow for automated, high-fidelity curation of cancer diagnosis and staging from unstructured patient records

Purpose Artificial intelligence (AI)-driven clinical document abstraction (CDA) using large language models (LLMs) is transforming oncology data management by extracting crucial data, such as staging and molecular profiles, from unstructured notes. We validated an agent that autonomously abstracts cancer diagnoses into structured, high-quality real-world evidence (RWE), accelerating research and supporting timely evidence-based treatment decisions with clinical-grade accuracy at scale and operational sustainability. Methods Our AI-CDA workflow uses a three-stage hybrid multi-agent AI design for high-accuracy, cost-efficient data abstraction: 1. Pre-screening: two specialized natural language processing (NLP) models fine-tuned on clinical notes scan the entire patient chart, classifying documents and identifying key cancer diagnosis events to select up to 100 relevant documents, reducing the LLM inference cost. 2. Extractor: a non-reasoning LLM (GPT-4.1) extracts all relevant diagnostic information from selected documents. 3. Structuring and normalization: a GPT-4.1 call synthesizes diagnosis summaries to produce structured clinical fields (e.g., diagnosis date, stage, histology) according to a predefined data model. Finally, structured fields are normalized to a fixed taxonomy by o3-mini. System performance was benchmarked against manual abstraction on a cohort of 1497 patients (499 breast, 499 lung, 499 pan-cancer) sampled from the Tempus RW database. Results The automated workflow performed strongly against manual abstraction. Tissue of origin abstraction achieved a micro F1-score of 0.94, with excellent performance on common cancers (e.g., lung: 0.98, breast: 0.98, colon: 0.97). Metastasis status detection achieved 0.92 F1-score, histology 0.91, and overall stage 0.83. Initial diagnosis date accuracy was 0.90. A subsequent, adjudicated re-evaluation of discordance revealed an even higher real-world performance by correcting initial manual labeling errors. Histology F1-score rose to 0.96 (+0.05) and overall staging rose to > 0.91 (a lift of over 10%). With regard to cost efficiency: pre-screening with two specialized NLP models reduced the average number of documents requiring LLM review >90% (from 266 to 24 per patient). This lowers LLM cost, mitigates performance degradation associated with longer context, and prioritizes only the most pertinent information. Conclusions This study demonstrates the feasibility and operational sustainability of an agentic AI-CDA workflow for highly accurate cancer data abstraction. This hybrid solution addresses three critical needs: scaling clinical research through automated, high-fidelity data abstraction; supporting timely evidence-based treatment decisions; and achieving operational sustainability by reducing inference cost and resourcing through a hybrid NLP/LLM workflow.