P-062 - ChatGPT for Migraine Patient Education: Ready for Prime Time?

Abstract Title: ChatGPT for Migraine Patient Education: Ready for Prime Time?

Background: Large Language Models (LLM) such as ChatGPT have recently demonstrated the ability to generate accurate and useful responses to clinical queries in some specialty areas such as orthopedics (1 Tharakan S et al) while having notable limitation in others areas such as neurosurgery (2 Gajjar et al). Currently, there are no prior studies have formally evaluated the accuracy of ChatGPT responses specifically pertaining to migraine management as rated by headache specialists.






Purpose/Objectives: This research project attempted to ascertain the completeness, accuracy and helpfulness of ChatGPT responses to common migraine treatment queries as rated by headache experts.

Methods: ChatGPT 4.0 (updated to April 2023) was queried in February, 2024 with 14 frequently asked questions about migraine treatment options with topics obtained from the American Migraine Foundation Migraine 101 page. Questions spanned the efficacy and safety of conventional and non-pharmacological options including medications for acute and preventative treatment, procedures, devices, behavioral, nutritional, physical therapy and self-care options. The un-edited answers for these queries were posed in a blinded fashion to 3 headache specialists from diverse background and training (1 UCNS certified physician, 1 physician assistant specializing in headache and 1 headache researcher). The experts independently rated the responses on a 1-5 Likert scale with 1= Completely Disagree to 5 = Completely Agree in 3 areas: 1. Completeness: the answer did not leave any pertinent information out 2. Accurateness: the answer did not provide any inaccurate or outdated information 3. Helpfulness: The answer provided easy to understand information for making safe and informed decisions. The raters also had the opportunity to provide comments on their rankings. Overall, the headache specialists ranked the ChatGPT responses to non-pharmacological therapies as being most complete, accurate and helpful including behavioral therapies (mean = 4.68) and self-care (mean = 4.67) with diet, exercise and physical therapy all achieving rates above 4.0. Questions related to safety of medication ranked the lowest (mean = 3.56) as did acute medications (mean = 3.67), procedures (mean = 3.78), and preventative medication (mean = 3.78). Review of qualitative comments related to lower ratings were related to ChatGPT responses which included outdated medication information, neglecting to mention newer medications (all approved prior to April 2023), incorrectly noting over the counter therapies as requiring a prescription, and not mentioning important safety information related to therapies during pregnancy.






Conclusions/Implications for future research and/or clinical care: While ChatGPT responses to frequently asked questions related to migraine therapy provided reasonably accurate, complete and helpful answers in the setting of non-pharmacological therapies, the same was not true of responses to prescriptions therapies and procedures where migraine specialists noted that important information on available therapies and safety was missing or inaccurate. Limitation of this pilot evaluation include lack of evaluation of headache specialists’ demographics and bias towards utilization of artificial intelligence as factors affecting their ratings. Future studies with more diverse input would be helpful to replicate these finding. In the meanwhile, clinicians should be aware of limitation to use of ChatGPT for patient education until further refinement and testing is completed.






References: 1 Tharakan S et al. Do ChatGPT and Google differ in answers to commonly asked patient questions regarding
total shoulder and total elbow arthroplasty? J Shoulder Elbow Surg. 2024 Jan 3.

2 Gajjar AA et al. Usefulness & Accuracy of AI Chatbot Responses to Patient Questions for Neurosurgical
Procedures. Neurosurgery. 2024 Feb 14