Accuracy of Artificial Intelligence Chatbots in Responding to Patient Questions About Anticoagulant Therapy: A Systematic Comparative Evidence Synthesis and Cross-Platform Evaluation Framework

Abstract

Background: Oral anticoagulants are high-risk medicines in which misunderstanding can contribute to bleeding, thrombosis, treatment interruption, or inappropriate self-management. Large language model chatbots can produce immediate conversational answers to patient questions, but the safety of using general medical performance as a proxy for anticoagulation-specific accuracy is uncertain.
Objective: To synthesize high-quality evidence available through 31 December 2023 regarding the accuracy, completeness, reliability, and safety of artificial intelligence chatbots for medical question answering and to translate that evidence into a reproducible cross-platform framework for anticoagulant patient questions.
Methods: A structured evidence synthesis was conducted using Q1 literature in medicine, cardiology, thrombosis, digital health, and artificial intelligence. Direct chatbot studies were combined with anticoagulation guidelines and patient-knowledge research. Because no qualifying pre-2024 Q1 study directly compared multiple public chatbots specifically on anticoagulant patient questions, no model-specific anticoagulation performance values were fabricated.
Results: In 2023 studies, ChatGPT responses were judged appropriate for 21 of 25 cardiovascular prevention questions, were preferred to physician responses in 78.6% of evaluations in a general patient-question study, and achieved a median physician-rated accuracy score of 5.5/6 across 284 medical questions. Med-PaLM achieved 92.6% alignment with scientific consensus on consumer medical questions, yet 5.9% of its answers were still judged potentially harmful. Other studies demonstrated specialty-specific inaccuracies, material response instability, and hallucinated references. Anticoagulant questions are especially sensitive to omissions involving missed doses, bleeding, drug interactions, procedures, renal function, adherence, and the difference between warfarin and direct oral anticoagulants.
Conclusion: General-purpose chatbots showed promising medical question-answering performance by the end of 2023, but the evidence did not support unsupervised anticoagulant counseling or a definitive ranking of public platforms. Cross-platform evaluation should use identical prompts, repeated generations, independent anticoagulation experts, guideline-based reference answers, and separate scoring of accuracy, completeness, readability, actionability, consistency, and potential harm.