Highlight
- A rule-based computable phenotype and a large language model (ChatGPT 4.1) were evaluated for identifying opioid use disorder (OUD) in emergency department (ED) EHR data.
- Both approaches demonstrated strong diagnostic accuracy with high negative predictive values and sensitivities around 80%.
- The large language model exhibited significantly higher specificity and a greater positive predictive value, suggesting fewer false-positive identifications.
- Prospective validation in diverse clinical settings is needed to confirm generalizability and integration potential into ED workflows.
Study Background
Opioid use disorder remains a critical public health crisis in the United States, with significant morbidity and mortality associated with opioid misuse, particularly in emergency department settings where many patients seek care for complications related to opioid use. Accurate identification of patients with OUD in EDs is essential for timely intervention, initiation of treatment protocols, and linking patients to addiction services. However, identification using electronic health records (EHRs) is complex due to variable documentation, undercoding, and stigma surrounding substance use diagnoses. Traditional computable phenotypes—rule-based algorithms that utilize structured data elements such as diagnosis codes, prescribed medications, toxicology screens, and clinical notes—have been used to systematically identify OUD. Despite their utility, these algorithms can miss cases or generate false positives due to rigid rules and incomplete data capture. Large language models (LLMs), like ChatGPT, leverage natural language processing and deep learning to interpret free-text clinical documentation. These models have potential to recognize nuanced clinical patterns not captured by structured codes. This study explores whether an LLM can enhance OUD detection in ED notes compared with a validated rule-based computable phenotype.Study Design
The authors conducted a retrospective cohort study at a single academic health system involving adult ED encounters from January 1, 2023, to October 17, 2024. A stratified random sample of 302 encounters was selected based on prior classification by an established rule-based computable phenotype for OUD. The phenotype incorporated several structured data components: ICD diagnosis codes, prescriptions for medications used in medication-assisted treatment (MAT), urine toxicology results, addiction medicine consults, and keyword searches in clinical notes. For comparison, the authors used ChatGPT 4.1 with zero-shot prompting to classify OUD status by analyzing free-text ED notes from the index visit. Two board-certified emergency physicians independently reviewed the full charts to serve as the reference standard, with a third reviewer resolving discrepancies. The primary outcomes were sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the phenotype and LLM against expert chart review. Inverse probability weighting based on sampling fractions adjusted prevalence estimates and performance metrics.Key Findings
Weighted prevalence of OUD in the sampled population was approximately 5.6% (95% CI, 4.0 to 7.0%). The structured computable phenotype demonstrated a sensitivity of 0.84 (95% CI, 0.42 to 0.97) and a specificity of 0.964 (95% CI, 0.96 to 0.97). Its positive predictive value was moderate at 0.58, with a high negative predictive value of 0.99. In comparison, the large language model (ChatGPT 4.1) had a sensitivity of 0.81 (95% CI, 0.70 to 0.88) and a notably higher specificity of 0.996 (95% CI, 0.993 to 0.998). This translated into a substantially higher positive predictive value of 0.92 while maintaining the negative predictive value of 0.99. Statistical testing confirmed the significantly superior specificity of the LLM (P < .0001). The findings imply that while the rule-based phenotype may capture slightly more true cases (higher sensitivity), it also produces more false positives, limiting PPV. Conversely, the LLM’s higher specificity reduces false-positive alerts, which is advantageous in the ED where alert fatigue can hinder clinical decision-making.Expert Commentary
This study represents an important advancement in leveraging artificial intelligence for substance use disorder detection in clinical settings. The combination of structured data and sophisticated natural language processing offers a promising avenue to improve case identification. Nonetheless, limitations include the single-center design and reliance on retrospective chart review, which may not capture undocumented clinical nuances. The relatively low prevalence of OUD in the selected cohort also affects predictive values. Moreover, the interpretability and transparency of large language models remain challenges, necessitating clinician trust and safeguards. Future investigations should focus on prospective validation across diverse healthcare systems and populations. Integration with clinical workflows, impact on patient outcomes, and cost-effectiveness analyses will be crucial for translation into practice.Conclusion
Both the rule-based computable phenotype and large language model approaches showed strong performance in identifying opioid use disorder from emergency department electronic health records. The structured phenotype demonstrated marginally better sensitivity, while the large language model achieved higher specificity and positive predictive value, potentially reducing false-positive alerts and improving clinical utility. Combining these methodologies or using LLMs for initial screening followed by rule-based confirmation could optimize accuracy and workflow efficiency. Further validation and assessment in real-world environments are necessary to harness these tools effectively to address the ongoing opioid crisis.Funding and ClinicalTrials.gov Registration
The published study did not specify funding sources or clinical trial registration numbers. Future research may benefit from multi-institutional collaborations and funding to facilitate wider implementation.References
1. Molina MF, Fenton C, LeSaint KT, Pimentel SD, Kohn MA, Kornblith AE. Computable Structured Phenotype Versus Large Language Model Identification of Opioid Use Disorder Using Electronic Health Record Data. Ann Emerg Med. 2026 Jun 13;88(5):562-569. doi:10.1016/j.annemergmed.2026.03.010. PMID: 42287282. 2. Volkow ND, McLellan TA. Opioid Abuse in Chronic Pain — Misconceptions and Mitigation Strategies. N Engl J Med. 2016 Mar 31;374(13):1253-63. doi:10.1056/NEJMra1507771. 3. Beam AL, Kohane IS. Big Data and Machine Learning in Health Care. JAMA. 2018 Apr 3;319(13):1317-1318. doi:10.1001/jama.2017.18391. 4. Kharrazi H, et al. Data quality in electronic health records: a critical appraisal. Yearb Med Inform. 2017 Aug;26(01):127-132. doi:10.15265/IY-2017-015. 5. Rajkomar A, et al. Scalable and accurate deep learning for electronic health records. npj Digital Medicine 1, Article number: 18 (2018). doi:10.1038/s41746-018-0029-1. 6. Sarker A, et al. Machine learning and natural language processing for identifying opioid misuse in electronic health records: a systematic review. J Med Internet Res. 2020 Nov 11;22(11):e18585. doi:10.2196/18585.Specialty calculators
For clinical reference; use professional judgment.
