Highlights
- Human reviewers show low sensitivity and moderate specificity in distinguishing AI-generated versus human-authored medical manuscripts.
- AI-generated personal statements for medical fellowship applications often surpass human-authored statements in readability and quality, influencing interview selection favorably.
- Frequent interaction with AI tools enhances reviewer ability to identify AI authorship, yet overall differentiation remains poor.
- Concerns regarding integrity and work ethic arise among program directors when AI-generated personal statements are suspected, highlighting ethical complexities.
Background
The integration of generative artificial intelligence (AI) technologies such as OpenAI’s ChatGPT into medical writing and application processes represents a transformative challenge in clinical academia. Medical manuscripts and personal statements are fundamental components of scholarly communication and trainee selection, respectively. Distinguishing AI-generated content from traditionally authored material is increasingly difficult, raising questions about reliability, ethical standards, and the need for updated guidelines. This review synthesizes recent evidence on human reviewers’ ability to detect AI authorship in medical manuscripts and personal statements, assesses the impact on evaluative outcomes, and considers implications for clinical education and editorial standards.
Key Content
Randomized Survey Study Assessing Human Reviewers’ Differentiation Ability
Helgeson et al. conducted a prospective randomized survey between October and December 2023 at a single academic center. Using ChatGPT 3.5, AI-generated medical manuscripts were created and randomized alongside human-authored manuscripts. Fifty-one physicians, ranging from post-doctorates to full professors, were blinded to manuscript origin and tasked with identifying authorship. Results demonstrated low sensitivity (31.2%, 95% CI 11.0–58.7%) and moderate specificity (55.6%, 95% CI 30.8–78.5%) in detecting AI-origin manuscripts. Positive and negative predictive values were similarly modest (38.5% and 47.6%, respectively). Notably, manuscripts from higher impact factor journals were identified with greater accuracy than lower-impact ones (P=0.037). Reviewer academic rank and prior peer-review experience did not predict accuracy; however, frequent use of AI tools significantly increased correct identification (OR up to 8.36, P<0.05). No manuscript quality metrics predicted accurate detection.
Comparative Analysis of AI-Generated vs Human-Authored Fellowship Personal Statements
Karakash et al. evaluated nine personal statements (four AI-generated via ChatGPT-4o focused on unique experiences, five human-authored) for spine surgery fellowship applications, reviewed by 8 blinded evaluators including attending surgeons and fellows. AI-generated statements outperformed in readability (mean score 65.69 vs. 56.40; P=0.016) and quality (63.00 vs. 51.80; P=0.004), while no significant differences were observed in originality or authenticity scores. Reviewers were unable to reliably distinguish AI from human authorship (P=1.000). Importantly, interview recommendation rates favored AI-generated statements markedly (84.4% vs. 62.5%, OR 3.24, P=0.045), suggesting that AI-authored statements may confer an evaluative advantage.
Perceptions of AI-Generated Personal Statements Among Obstetric Anesthesia Fellowship Directors
Ruiz et al. surveyed U.S. obstetric anesthesia fellowship program directors evaluating four personal statements (two AI-generated by ChatGPT, two human-written). Directors could not accurately identify AI-generated content and rated AI statements higher in readability and originality. Despite this, a majority expressed moderate to extreme concern regarding applicants’ integrity, work ethic, and reliability if AI authorship was suspected. This ambivalence underscores a tension between recognizing AI’s capability to enhance writing quality and ethical reservations about its use. The study advocates for explicit programmatic policies addressing AI use in applications.
Synthesis of Findings Across Studies
Collectively, these investigations reveal convergent themes: (1) AI-generated medical manuscripts and personal statements are often indistinguishable to human reviewers; (2) AI-generated content may surpass human-written counterparts in certain quality metrics; (3) familiarity with AI tools enhances detection ability but does not eliminate misclassification; (4) ethical concerns and policy gaps remain problematic in academic and application contexts.
Table 1 summarizes comparative metrics across the studies:
| Study | Sample | Key Outcomes | Reviewer Detection Accuracy | AI Content Quality | Ethical Impact |
|---|---|---|---|---|---|
| Helgeson et al. (2025) | 51 physicians, 3 manuscripts each | Sensitivity: 31.2%; Specificity: 55.6% | Low accuracy in distinguishing AI vs human manuscripts | Comparable to human manuscripts | N/A |
| Karakash et al. (2025) | 9 personal statements, 8 reviewers | Interview recommendations favor AI-generated (84.4% vs 62.5%) | Non-significant difference (P=1.000) | Higher readability and quality | AI statements more favorably rated but ethics unaddressed |
| Ruiz et al. (2025) | 4 personal statements, survey of program directors | AI statements rated more readable and original | Unable to distinguish authorship | Rated favorably on quality | Concerns about integrity and work ethic if AI suspected |
Expert Commentary
The rapid advancement of generative AI calls for critical reexamination of academic authorship norms and evaluation frameworks. The consistent failure of human experts to reliably discern AI-generated content highlights the sophistication of AI language models and their potential to blur boundaries between human and machine authorship. This phenomenon poses challenges for peer review, academic integrity, and admissions processes, given AI’s capacity to enhance text quality and readability beyond many human drafts.
While AI may provide equitable access to high-caliber editing and composition, especially for non-native English speakers or applicants with limited writing skills, it also raises questions about originality and ethical transparency. The ethical dilemma resides in balancing the benefits of AI assistance with principles of authentic authorship and fairness. Current editorial and training guidelines rarely address explicit AI disclosure, generating ambiguity.
Frequent users of AI demonstrated better detection capabilities, possibly due to familiarity with AI linguistic patterns; however, widespread expertise remains lacking. This suggests a training gap for clinicians and editors tasked with manuscript and application review. Strategic incorporation of AI literacy into academic curricula and reviewer training may be warranted.
Future policies should consider establishing clear standards regarding AI use, transparency mandates, and the development of algorithmic detectors complementing human judgment. Moreover, academic institutions and program directors should offer guidance delineating acceptable AI integration in application materials to uphold fairness while embracing technological advancements.
Conclusion
The evidence underscores that AI-generated medical manuscripts and fellowship personal statements are effectively indistinguishable from human-authored texts by most professional reviewers. AI often enhances document quality, influencing evaluative outcomes positively. This trend necessitates urgent dialogue and policy development within medical education and publishing to address ethical, practical, and educational implications. Ongoing research should optimize detection methods, clarify AI authorship roles, and frame ethical frameworks supporting responsible AI use in academic and clinical contexts.
References
- Helgeson SA, Johnson PW, Gopikrishnan N, et al. Human Reviewers’ Ability to Differentiate Human-Authored or Artificial Intelligence-Generated Medical Manuscripts: A Randomized Survey Study. Mayo Clin Proc. 2025 Apr;100(4):622-633. doi:10.1016/j.mayocp.2024.08.029 IF: 6.7 Q1 . PMID:40057868 IF: 6.7 Q1 .
- Karakash WJ, Avetisian H, Ragheb JM, et al. Artificial Intelligence vs Human Authorship in Spine Surgery Fellowship Personal Statements: Can ChatGPT Outperform Applicants? Global Spine J. 2025 May 20:21925682251344248. doi:10.1177/21925682251344248 IF: 3.0 Q1 . Epub ahead of print. PMID:40392947 IF: 3.0 Q1 ; PMCID:PMC12092409 IF: 3.0 Q1 .
- Ruiz AM, Kraus MB, Arendt KW, et al. Artificial intelligence-created personal statements compared with applicant-written personal statements: a survey of obstetric anesthesia fellowship program directors in the United States. Int J Obstet Anesth. 2025 Feb;61:104293. doi:10.1016/j.ijoa.2024.104293 IF: 2.3 Q2 . Epub 2024 Nov 15. PMID:39591877 IF: 2.3 Q2 .
