Developing prompts from large language model for extracting clinical information from pathology and ultrasound reports in breast cancer
Hyeon Seok Choi, Jun Yeong Song, Kyung Hwan Shin et al. · Radiation Oncology Journal · 2023
PURPOSE: We aimed to evaluate the time and cost of developing prompts using large language model (LLM), tailored to extract clinical factors in breast cancer patients and their accuracy. MATERIALS AND METHODS: We collected data from reports of surgical pathology and ultrasound from breast cancer patients who underwent radiotherapy from 2020 to 2022. We extracted the information using the Generative Pre-trained Transformer (GPT) for Sheets and Docs extension plugin and termed this the "LLM" method. The time and cost of developing the prompts with LLM methods were assessed and compared with those spent on collecting information with "full manual" and "LLM-assisted manual" methods. To assess accuracy, 340 patients were randomly selected, and the extracted information by LLM method were compared with those collected by "full manual" method. RESULTS: Data from 2,931 patients were collected. We developed 12 prompts for Extract function and 12 for Format function to extract and standardize the information. The overall accuracy was 87.7%. For lymphovascular invasion, it was 98.2%. Developing and processing the prompts took 3.5 hours and 15 minutes, respectively. Utilizing the ChatGPT application programming interface cost US $65.8 and when factoring in the estimated wage, the total cost was US $95.4. In an estimated comparison, "LLM-assisted manual" and "LLM" methods were time- and cost-efficient compared to the "full manual" method. CONCLUSION: Developing and facilitating prompts for LLM to derive clinical factors was efficient to extract crucial information from huge medical records. This study demonstrated the potential of the application of natural language processing using LLM model in breast cancer patients. Prompts from the current study can be re-used for other research to collect clinical information.
Cited by 132 works. doi:10.3857/roj.2023.00633 Open-access PDF
Authors
References
- Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
- Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
- GPT-3: Its Nature, Scope, Limits, and Consequences
- Artificial intelligence approaches using natural language processing to advance EHR-based clinical research
- Global Mental Health Services and the Impact of Artificial Intelligence–Powered Large Language Models
- Clinical Natural Language Processing for Radiation Oncology: A Review and Practical Primer
- Incorporating Risk Factors to Identify the Indication of Post-mastectomy Radiotherapy in N1 Breast Cancer Treated with Optimal Systemic Therapy: A Multicenter Analysis in Korea (KROG 14-23)
- Changes in the working conditions and learning environment of medical residents after the enactment of the Medical Resident Act in Korea in 2015: a national 4-year longitudinal study
- Identification of Risk Factors for Locoregional Recurrence in Breast Cancer Patients with Nodal Stage N0 and N1: Who Could Benefit from Post-Mastectomy Radiotherapy?
- Patients with N1 breast cancer: Who could benefit from supraclavicular fossa radiotherapy?