Large Language Models for Generating Patient Education Materials on Type 2 Diabetes: Comparative Assessment of Readability, Accuracy, Completeness, and Hallucination Risk

Abstract

Background: Type 2 diabetes requires sustained self-management education covering medication use, nutrition, glucose monitoring, physical activity, complication prevention, and responses to abnormal glucose values. Large language models (LLMs) can generate fluent patient-facing explanations within seconds, but their suitability cannot be judged by factual accuracy alone.
Objective: To synthesize high-quality evidence available through 31 December 2023 concerning the ability of LLMs to generate type 2 diabetes patient education, focusing on readability, factual accuracy, completeness, reproducibility, and hallucination or unsupported-content risk.
Methods: A structured comparative evidence synthesis was conducted using Q1 peer-reviewed literature published no later than 31 December 2023. Diabetes-specific LLM studies were prioritized and supplemented by high-quality general medical evaluations where they directly informed assessment of accuracy, completeness, patient-facing communication, or safety. Heterogeneous tasks and scoring systems precluded meta-analysis; results were therefore synthesized by prespecified quality domain.
Results: Diabetes-specific evaluations showed high but nonuniform factual performance. ChatGPT answered all 24 items of the Diabetes Knowledge Questionnaire correctly in one study, while another diabetes evaluation reported very high expert accuracy scores but a mean Flesch-Kincaid grade level of 13.8, indicating substantial reading burden. In a Turing-test-inspired diabetes study, two of ten ChatGPT answers contained factual errors, including one potentially consequential error concerning exercise and glucose response. An AI dietitian study received favorable professional ratings for 162 of 168 generated responses. Across studies, accuracy frequently coexisted with excessive complexity, omissions, lack of source transparency, and occasional incorrect statements.
Conclusion: LLMs can produce useful first drafts of type 2 diabetes education, but patient-facing deployment requires a multidimensional quality gate. Accuracy, completeness, readability, and hallucination risk should be evaluated separately, with clinician review, guideline verification, version documentation, and post-release monitoring. A highly accurate response is not necessarily understandable, complete, current, or safe.