This blinded expert-rated study compared 11 AI systems—including nine general-purpose large language models and two retrieval-augmented platforms—using standardized synthetic periodontal clinical vignettes. Retrieval-augmented systems, notably Perplexity and OpenEvidence, outperformed general-purpose LLMs in accuracy and safety metrics. Despite promising average scores, a concerning proportion of AI-generated responses contained potentially dangerous information underscoring the need for cautious clinical use as assistive tools, not replacements for specialist judgment.
Study Background
Periodontal diseases represent a significant global public health concern due to their chronic nature, association with systemic conditions, and impact on oral health-related quality of life. Accurate diagnosis and management hinge upon complex clinical assessment, which can benefit from advanced decision-support tools. Artificial intelligence (AI), particularly large language models (LLMs) and retrieval-augmented platforms, has shown promise in medical applications by synthesizing knowledge and supporting clinical reasoning. However, for periodontal diseases, a comparative evaluation of these AI systems remains sparse. This study addresses the critical need to assess AI accuracy, safety, and reliability specifically in periodontal diagnosis and management to inform future clinical integration.
Study Design
The investigation employed a blinded, randomized block design evaluating 11 AI systems comprising nine general-purpose LLMs (e.g., Claude 4.7 Opus), one general-purpose retrieval-augmented platform (Perplexity), and one medical-domain retrieval-augmented platform (OpenEvidence). Thirty standardized synthetic periodontal clinical vignettes, representing diverse scenarios—acute, chronic, and complex conditions—were presented to each AI for response generation. Responses were independently scored by six blinded expert periodontists using a validated 5-point Likert scale assessing four domains: accuracy of information, safety (absence of harmful recommendations), freedom from hallucinations (inaccurate or fabricated content), and completeness of answers.
Statistical analyses included Friedman and Conover–Iman tests with Holm correction for multiple comparisons, supplemented by mixed-effects and ordinal models for sensitivity analyses. The study aimed to discern differences in performance metrics across the AI platforms and identify those with optimal clinical applicability.
Key Findings
Overall, the AI systems demonstrated mean composite scores ranging from 3.28 to 4.86 on the 5-point scale, surpassing the midpoint benchmark of 3.0, indicating generally acceptable performance levels. However, a notable and clinically significant finding was that between 3.3% and 46.7% of individual AI responses scored in the “dangerous” or potentially harmful category (scores ≤ 2), varying by platform. Such variability highlights a credible risk of misleading recommendations if these tools are used unchecked.
Statistical testing established significant performance differences among the AI systems (p < 0.001). Kendall’s W coefficient indicated a small-to-medium overall effect size (W = 0.17), reflecting moderated consistency across systems. Yet, within subcategories such as chronic or complex periodontal scenarios, the effect size was substantially larger (up to W = 0.82), underscoring distinctive platform abilities dependent on clinical complexity.
Among the platforms, the general-purpose retrieval-augmented system Perplexity, the medical-domain retrieval-augmented platform OpenEvidence, and the LLM Claude 4.7 Opus emerged as the top performers across all domains, especially excelling in safety and accuracy. The apparent superior performance of retrieval-augmented systems was partially confounded by their characteristically longer, more comprehensive responses.
Despite promising findings, the wide range in safety scores and presence of hallucinations warns against uncritical, undifferentiated deployment of these AI tools in clinical periodontal practice without robust expert supervision.
Expert Commentary
This study is pioneering in benchmarking AI tools specifically for periodontal disease clinical reasoning—a domain that blends symptom assessment, radiographic interpretation, and nuanced treatment planning. The blinded expert rating design is a methodological strength, reducing bias and emulating real-world specialist evaluation. The inclusion of retrieval-augmented systems is innovative, reflecting the evolving landscape of AI that combines language models with external knowledge bases, seemingly enhancing factual reliability.
However, the influence of response length on quality scorings raises methodological questions. Longer answers may provide more information, potentially inflating completeness and safety ratings, but also risk verbosity or irrelevant content, potentially hindering practical clinical use. Moreover, although synthetic vignettes standardize case presentation, they cannot encompass the full complexity or unpredictability of real-patient encounters, leaving open questions regarding external validity.
The findings reinforce the consensus emerging from AI in healthcare literature: these systems should serve as decision-support adjuncts rather than substitutes for expert clinicians. Close integration with clinical expertise, critical appraisal, and possibly continuous AI training on domain-specific data are imperative before widespread adoption.
Conclusion
This comparative evaluation of 11 AI platforms for periodontal disease diagnosis and management reveals that retrieval-augmented systems currently outperform general-purpose LLMs in expert-rated accuracy and safety. Nonetheless, the notable presence of dangerous or hallucinated responses cautions against unmonitored clinical application. These AI tools hold promise as adjunctive supports for specialist judgement but are not substitutes. Future research should refine retrieval-augmentation approaches, optimize output conciseness, and validate performance in real-world clinical settings to safely harness AI’s potential in periodontology.
Funding and Clinical Trials
The study was not indicated to have external funding sources. No clinical trial registration was reported.
References
1. Mayer Y, Demetriou B, Rasperini G, et al. Large Language Models and Retrieval-Augmented Platforms for the Diagnosis and Management of Periodontal Diseases: A Blinded Expert-Rated Comparative Study of 11 Systems. J Clin Periodontol. 2026 Jul;53(10):1546-1556. PMID: 42402252.
By
No Comments
