Evaluating large language models as an educational tool for meningioma patients: patient and clinician perspectives
Diana-Coralia Dehelean, Sebastian H. Maier, Alev Altay-Langguth et al. · Radiation Oncology · 2025
BACKGROUND: The study explores the potential of ChatGPT, an advanced large language model (LLM) by OpenAI, in educating patients about meningioma, a common type of brain tumor. While ChatGPT has generated significant debate regarding its utility and ethics, its growing popularity suggests that patients may increasingly use such tools for medical information. The study specifically examines how patients who have undergone radiation therapy for meningioma perceive the information generated by ChatGPT, integrating both patient feedback and clinical assessment. METHODS: Eight meningioma-related questions on diagnosis, treatment options, and radiation therapy were posed to ChatGPT 4. A questionnaire with these responses and feedback items was developed to assess utility, accuracy, clarity, and alignment with patients' experiences. Nine clinicians first rated each response's relevance, correctness, and completeness on a five-point Likert scale. Subsequently, 28 patients with meningioma completed the questionnaire during their first follow-up visit (three months post-radiation therapy). Finally, the same questions were presented to three other large language models (ChatGPT 4o mini, Gemini Free, Gemini Advanced), and seven blinded clinicians rated each model's responses before selecting the most accurate, eloquent, and comprehensive overall. RESULTS: The study cohort included 28 meningioma patients, mostly female, with a median age of 60 years. Most patients found the information clear, accurate, and reflective of their experiences, with 60% willing to use ChatGPT for future inquiries. Clinicians rated the relevance and correctness of the information highly, although completeness was rated slightly lower, particularly for questions about specific radiation therapy details and side effects. ChatGPT 4 and its newer version ChatGPT 4o mini received the highest, nearly identical scores among the four LLMs evaluated, while Gemini Free scored the lowest in clinician assessments. CONCLUSIONS: ChatGPT demonstrates potential as a supplementary educational tool for meningioma patients, though some areas may require improvement, particularly in providing comprehensive information. The study highlights the potential for integrating AI in patient education, while also noting the need for clinical oversight to ensure accuracy and completeness. TRIAL REGISTRATION: LMU ethics vote nr.: 23-0742.
Cited by 8 works. doi:10.1186/s13014-025-02671-2 Open-access PDF
Authors
- Diana-Coralia Dehelean
- Sebastian H. Maier
- Alev Altay-Langguth
- Alexander Nitschmann
- Michael Schmeling
- Daniel F. Fleischmann
- Paul Rogowski
- Christian Trapp
- Stefanie Corradini
- Claus Belka
- Stephan Schönecker
- Sebastian Norbert Marschner
References
- Decision aids for people facing health treatment or screening decisions
- Artificial Hallucinations in ChatGPT: Implications in Scientific Writing
- ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations
- Assessing the Accuracy and Reliability of AI-Generated Medical Responses: An Evaluation of the Chat-GPT Model
- Is ChatGPT accurate and reliable in answering questions regarding head and neck cancer?
- ChatGPT in glioma adjuvant therapy decision making: ready to assume the role of a doctor in the tumour board?
- Can ChatGPT, an Artificial Intelligence Language Model, Provide Accurate and High-quality Patient Information on Prostate Cancer?
- Evaluating the Efficacy of ChatGPT as a Patient Education Tool in Prostate Cancer: Multimetric Assessment
- Meningioma: International Consortium on Meningiomas consensus review on scientific advances and treatment paradigms for clinicians, researchers, and patients
- Assessing ChatGPT’s ability to answer questions pertaining to erectile dysfunction: can our patients trust it?