Highlight
AI-enabled medical devices have rapidly expanded, especially in radiology and cardiovascular specialties, but only about half present clinical performance data at FDA approval.
Less than one-third of clinical studies report sex-specific or age-specific data, limiting insights on device applicability across diverse populations.
Retrospective designs dominate device validation, with prospective and randomized trials remaining rare, raising questions about robustness of evidence.
The scarcity of detailed development and performance data highlights significant gaps in assessing the clinical generalizability of these devices.
Study Background and Disease Burden
Artificial intelligence (AI) is increasingly integrated into medical devices with the promise of enhancing diagnostic accuracy, prognostication, and therapeutic guidance. The FDA has recognized and approved hundreds of AI-enabled medical devices over the recent decade, reflecting broad clinical interest and technological advancements. These devices predominantly target high-impact clinical domains such as radiology—where image interpretation is pivotal—as well as cardiovascular and neurologic conditions, where timely decision-making can significantly influence outcomes.
Despite this growth, the broader clinical generalizability—meaning the ability of these devices to perform safely and effectively across a wide range of patient populations and real-world settings—remains uncertain. Generalizability is essential to ensure equitable health care delivery and to prevent harms from biased or inaccurate AI models. Moreover, given the complexity of AI algorithms and their development, rigorous clinical validation studies are vital but may be lacking, especially regarding demographic inclusivity and prospective evaluation.
Study Design
This cross-sectional study analyzed all AI-enabled medical devices approved by the US Food and Drug Administration (FDA) and publicly listed as of August 31, 2024. Data extracted included device specialty, type (software only vs. implantable), and the presence of clinical evaluation data reported in the FDA summaries.
The main endpoints assessed were the extent and design of clinical performance studies supporting device approval, the reporting of discriminatory performance metrics such as sensitivity, specificity, and area under the curve (AUC), and the inclusion of age- and sex-specific subgroup data critical for assessing generalizability.
Key Findings
A total of 903 AI-enabled medical devices were included, primarily in radiology (76.6%), cardiovascular medicine (10.1%), and neurology (3.2%). The majority were software-only devices (73.5%), with only a small number being implantable (0.7%). Notably, detailed descriptions of device development, including training data and algorithmic design, were largely absent from FDA public summaries.
Clinical performance studies were documented for 505 devices (55.9%), whereas 218 devices (24.1%) explicitly reported no performance studies. Among these studies, retrospective designs were most common (38.2%), with prospective studies constituting only 8.1% and randomized controlled studies a mere 2.4%. This raises concerns about the robustness and reliability of evidentiary bases for many devices.
Discriminatory performance metrics were reported for less than a quarter of devices, with sensitivity noted for 36.2%, specificity for 34.9%, and AUC for only 16.2%. These metrics provide critical insight into diagnostic accuracy but remain underreported in publicly available summaries.
Equally important, demographic granularity in performance data was limited. Only 28.7% of clinical studies reported sex-specific outcomes, and 23.2% addressed age-related subgroups. This shortfall impedes understanding of how well AI devices perform across different patient demographics, a key factor for clinical generalizability.
Expert Commentary
The findings underscore a significant gap between the rapid proliferation of AI-enabled medical devices and the quality and transparency of clinical evidence supporting their use. Retrospective studies dominate, which, while informative, are less rigorous than prospective or randomized trials necessary for confirming efficacy and safety. The paucity of demographic subgroup analyses raises equity concerns; without this data, devices risk underperforming or misdiagnosing in underrepresented populations.
Moreover, the absence of detailed methodological data in publicly accessible FDA summaries limits clinicians’ ability to critically appraise devices prior to adoption. Dr. GCM Siontis, a co-author, emphasizes the importance of “ongoing monitoring and regular re-evaluation to identify and address unexpected performance changes during widespread clinical use,” highlighting that regulatory approval is not the end point but the beginning of continuous assessment.
These challenges align with broader calls for more rigorous standards in AI medical device evaluation, including the adoption of prospective trial designs, transparent reporting frameworks, and active surveillance post-approval. Addressing these deficiencies is critical to ensure AI technologies enhance rather than jeopardize patient care.
Conclusion
This comprehensive analysis reveals that although AI-enabled medical devices are swiftly gaining regulatory approval, significant limitations in the clinical evidence base and reporting standards limit their clinical generalizability. More than half lacked prospective or randomized evaluation, and demographic subgroup data were infrequently reported.
Moving forward, robust clinical validation through prospective and randomized studies, alongside transparent and inclusive reporting of demographic data, is essential. Such measures will better safeguard effective and equitable use of AI medical devices across diverse patient populations. Clinicians and regulators must emphasize continuous postmarket surveillance to promptly detect and mitigate any performance degradation or biases.
This study highlights an urgent need to balance innovation with rigorous evidence to fully realize the potential of AI in clinical medicine while minimizing risks associated with premature or inadequately validated technology adoption.
References
1. Windecker D, Baj G, Shiri I, Kazaj PM, Kaesmacher J, Gräni C, Siontis GCM. Generalizability of FDA-Approved AI-Enabled Medical Devices for Clinical Use. JAMA Netw Open. 2025 Apr 1;8(4):e258052. doi:10.1001/jamanetworkopen.2025.8052. PMID:40305017; PMCID:PMC12044510.
2. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019 Jan;25(1):44-56. doi:10.1038/s41591-018-0300-7.
3. Amann J, Blasimme A, Vayena E, Frey D, Madai VI. Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC Med Inform Decis Mak. 2020 Oct 20;20(1):310. doi:10.1186/s12911-020-01332-6.
4. FDA. Artificial Intelligence and Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan. FDA.gov. 2021.
5. Gottesman O, Johansson F, Komorowski M, Faisal AA, Sontag D, Doshi-Velez F, Celi LA, Badawi O. Guidelines for Reinforcement Learning in Healthcare. Nat Med. 2019 Jan;25(1):16-18. doi:10.1038/s41591-018-0342-5.
