AIMS: The aim of this study was to compare the clinical decision-making for benzodiazepine deprescribing between a healthcare provider (HCP) and an artificial intelligence (AI) chatbot GPT4 (ChatGPT-4). METHODS: We analysed real-world data from a Croatian cohort of community-dwelling benzodiazepine patients (n = 154) within the EuroAgeism H2020 ESR 7 project. HCPs evaluated the data using pre-established deprescribing criteria to assess benzodiazepine discontinuation potential. The research team devised and tested AI prompts to ensure consistency with HCP judgements. An independent researcher employed ChatGPT-4 with predetermined prompts to simulate clinical decisions for each patient case. Data derived from human-HCP and ChatGPT-4 decisions were compared for agreement rates and Cohen's kappa. RESULTS: Both HPC and ChatGPT identified patients for benzodiazepine deprescribing (96.1% and 89.6%, respectively), showing an agreement rate of 95% (κ = .200, P = .012). Agreement on four deprescribing criteria ranged from 74.7% to 91.3% (lack of indication κ = .352, P < .001; prolonged use κ = .088, P = .280; safety concerns κ = .123, P = .006; incorrect dosage κ = .264, P = .001). Important limitations of GPT-4 responses were identified, including 22.1% ambiguous outputs, generic answers and inaccuracies, posing inappropriate decision-making risks. CONCLUSIONS: While AI-HCP agreement is substantial, sole AI reliance poses a risk for unsuitable clinical decision-making. This study's findings reveal both strengths and areas for enhancement of ChatGPT-4 in the deprescribing recommendations within a real-world sample. Our study underscores the need for additional research on chatbot functionality in patient therapy decision-making, further fostering the advancement of AI for optimal performance.

Center for Applied Pharmacy Faculty of Pharmacy and Biochemistry University of Zagreb Zagreb Croatia

City Pharmacy Zagreb Zagreb Croatia

Department of Geriatrics and Gerontology 1st Faculty of Medicine Prague Charles University Prague Czech Republic

Department of Social and Clinical Pharmacy Faculty of Pharmacy in Hradec Králové Charles University Hradec Králové Czech Republic

Department of Social Pharmacy and Pharmaceutical Legislation Faculty of Pharmacy University of Belgrade Belgrade Serbia

See more in PubMed

Liu S, Wright APA, Patterson BL, et al. Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Informatics Assoc. 2023;30(7):1237-1245. doi:10.1093/jamia/ocad072

Giordano C, Brennan M, Mohamed B, Rashidi P, Modave F, Tighe P. Accessing artificial intelligence for clinical decision-making. Front Digit Heal. 2021;25(3):645232.

Markota M, Rummans TA, Bostwick JM, Lapid MI. Benzodiazepine use in older adults: dangers, management, and alternative therapies. Mayo Clin Proc. 2016;91(11):1632-1639. doi:10.1016/j.mayocp.2016.07.024

Gupta A, Bhattacharya G, Balaram K, Tampi D, Tampi RR. Benzodiazepine use among older adults. Neurodegener Dis Manag. 2021;11(1):5-8. doi:10.2217/nmt-2020-0056

Reeve E, Gnjidic D, Long J, Hilmer S. A systematic review of the emerging definition of ‘deprescribing’ with network analysis: implications for future research and clinical practice. Br J Clin Pharmacol. 2015;80(6):1254-1268. doi:10.1111/bcp.12732

Okeowo DA, Zaidi STR, Fylan B, Alldred DP. Barriers and facilitators of implementing proactive deprescribing within primary care: a systematic review. Int J Pharm Pract. 2023;31(2):126-152. doi:10.1093/ijpp/riad001

Evrard P, Damiaens A, Patey AM, Grimshaw JM, Spinewine A. Barriers and enablers towards benzodiazepine-receptor agonists deprescribing in nursing homes: a qualitative study of stakeholder groups. Explor Res Clin Soc Pharm. 2023;9:100258. doi:10.1016/j.rcsop.2023.100258

Reeve E. Deprescribing tools: a review of the types of tools available to aid deprescribing in clinical practice. J Pharm Pract Res. 2020;50(1):98-107. doi:10.1002/jppr.1626

O'Mahony D, Cherubini A, Guiteras AR, et al. STOPP/START criteria for potentially inappropriate prescribing in older people: version 3. Eur Geriatr Med. 2023;31(1):1-8.

Primary Health Tasmania. A guide to benzodiazepines recommended deprescribing strategy. 2019. https://www.primaryhealthtas.com.au/wp-content/uploads/2023/03/A-guide-to-deprescribing-benzodiazepines.pdf. Accessed November 18, 2023.

American Geriatrics Society Beers Criteria® Update Expert Panel. American Geriatrics Society 2023 updated AGS Beers Criteria® for potentially inappropriate medication use in older adults. J Am Geriatr Soc. 2023;71(7):2052-2081.

Royal Australian College of General Practitioners. Evidence-based guidance for benzodiazepines. In Prescribing drugs of dependence in general practice. https://www.racgp.org.au/clinical-resources/clinical-guidelines/key-racgp-guidelines/view-all-racgp-guidelines/drugs-of-dependence/part-b/evidence-based-guidance-for-benzodiazepines. Accessed February 22, 2023.

Mann N-K, Mathes T, Sönnichsen A, et al. Potentially inadequate medications in the elderly: PRISCUS 2.0-first update of the PRISCUS list. Dtsch Arztebl Int. 2023;120(1-2):3-10.

Pazan F, Weiss C, Wehling M. The EURO-FORTA (Fit fOR The Aged) list: international consensus validation of a clinical tool for improved drug treatment in older people. Drugs Aging. 2018;35(1):61-71. doi:10.1007/s40266-017-0514-2

Evrard P, Pétein C, Beuscart JB, Spinewine A. Barriers and enablers for deprescribing benzodiazepine receptor agonists in older adults: a systematic review of qualitative and quantitative studies using the theoretical domains framework. Implement Sci. 2022;17(1):41. doi:10.1186/s13012-022-01206-7

Niznik JD, Ferreri SP, Armistead LT, et al. Primary-care prescribers' perspectives on deprescribing opioids and benzodiazepines in older adults. Drugs Aging. 2022;39(9):739-748. doi:10.1007/s40266-022-00967-6

Levivien C, Cavagna P, Grah A, et al. Assessment of a hybrid decision support system using machine learning with artificial intelligence to safely rule out prescriptions from medication review in daily practice. Int J Clin Pharmacol. 2022;44(2):459-465. doi:10.1007/s11096-021-01366-4

Damiani G, Altamura G, Zedda M, et al. Original research: potentiality of algorithms and artificial intelligence adoption to improve medication management in primary care: a systematic review. BMJ Open. 2023;13(3):65301. doi:10.1136/bmjopen-2022-065301

Walker LE, Abuzour AS, Bollegala D, et al. The DynAIRx Project Protocol: Artificial Intelligence for dynamic prescribing optimisation and care integration in multimorbidity. J Multimorb Comorb. 2022;12:263355652211454. doi:10.1177/26335565221145493

OpenAI. ChatGPT-4 [Internet]. 2023. https://openai.com/research/gpt-4. Accessed August 3, 2023.

NSW Therapeutic Advisory Group Inc. Deprescribing guide for benzodiazepines and Z drugs. 2018. https://www.nswtag.org.au/wp-content/uploads/2018/06/1.1-Deprescribing-Guide-for-Benzodiazepines-and-Z-Drugs.pdf. Accessed November 18, 2023.

Pottie K, Thompson W, Davies S, et al. Deprescribing benzodiazepine receptor agonists: evidence-based clinical practice guideline. Can Fam Physician. 2018;64(5):339-351.

Primary Health Tasmania. A guide to the use of benzodiazepines in older adults. 2016. https://www.primaryhealthtas.com.au/wp-content/uploads/2018/09/A-guide-to-the-use-of-benzodiazapenes-in-older-people.pdf. Accessed November 18, 2023.

Lee JY, Farrell B, Holbrook AM. Deprescribing benzodiazepine receptor agonists taken for insomnia: a review and key messages from practice guidelines. Pol Arch Intern Med. 2019;129(1):43-49. doi:10.20452/pamw.4391

McHugh ML. Interrater reliability: the kappa statistic. Biochem Med. 2012;22(3):276-282. doi:10.11613/BM.2012.031

OpenAI. ChatGPT-4 technical report. ArXiv. 2023. doi:10.48550/arXiv.2303.08774

Gilson A, Safranek CW, Huang T, et al. How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023;9:e45312. doi:10.2196/45312

Strong E, DiGiammarino A, Weng Y, et al. Chatbot vs medical student performance on free-response clinical reasoning examinations. JAMA Intern Med. 2023;183(9):1028-1030.

Ayers JW, Poliak A, Dredze M, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med. 2023;183(6):589-596. doi:10.1001/jamainternmed.2023.1838

Page AT, Etherton-Beer CD, Clifford RM, Burrows S, Eames M, Potter K. Deprescribing in frail older people-do doctors and pharmacists agree? Res Soc Adm Pharm. 2016;12(3):438-449. doi:10.1016/j.sapharm.2015.08.011

Gravel J, D'Amours-Gravel M, Osmanlliu E. Learning to fake it: limited responses and fabricated references provided by ChatGPT for medical questions. Mayo Clin Proc Digit Heal. 2023;1(3):226-234. doi:10.1016/j.mcpdig.2023.05.004

Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell. 2023;6:1169595. doi:10.3389/frai.2023.1169595

Lee P, Bubeck S, Petro J. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. N Engl J Med. 2023;388(13):1233-1239. doi:10.1056/NEJMsr2214184

Yeung JA, Kraljevic Z, Luintel A, et al. AI chatbots not yet ready for clinical use. Front Digit Heal. 2023;5:1161098. doi:10.3389/fdgth.2023.1161098