Live Versus Delayed Video-Based TEAM Assessment in Real-World Cardiac Arrest: Insights into Reliability and Rater Experience

Live Versus Delayed Video-Based TEAM Assessment in Real-World Cardiac Arrest: Insights into Reliability and Rater Experience

Highlight

  • Delayed video-based assessments of nontechnical emergency team skills yield comparable average scores to live assessments in cardiac arrest resuscitations.
  • Event-level agreement between live and video assessments is limited, indicating variability in judging specific cases.
  • Senior, more experienced raters demonstrate higher concordance between live and video reviews.
  • Video review remains valuable for team debriefing and monitoring but requires experienced raters for high-stakes appraisal of individual events.

Study Background

Effective teamwork and nontechnical skills, including communication, leadership, and task management, are critical determinants of success in high-stakes emergency scenarios such as adult cardiac arrest resuscitations. The Team Emergency Assessment Measure (TEAM) is a validated tool designed to quantify these nontechnical skills and thus serve as a metric for quality improvement and training.

While live assessment captures real-time team dynamics, it may be subject to recall bias and observational constraints. Video-based review offers the possibility to assess performance retrospectively with opportunities for repeated viewings, but its validity and reliability compared to live scoring in real-world resuscitations remain uncertain. Understanding the concordance between these two methods and the influence of rater experience has significant implications for clinical quality assurance and research.

Study Design

This prospective observational study was conducted in 2024 at a tertiary emergency department in Taiwan. Adult cardiac arrest resuscitations were video-recorded as part of routine quality improvement measures. For each event, one trained physician or nurse who directly participated in the resuscitation performed an immediate, live TEAM assessment directly following the event. Approximately 30 days later, the same rater re-assessed the identical resuscitation from video recordings, blinded to their initial ratings, to measure the consistency and agreement between live and delayed video ratings.

The primary endpoint was agreement between live and video TEAM scores, including total TEAM scores and global ratings. Agreement metrics included intraclass correlation coefficients (ICC) and Bland-Altman analyses. An exploratory subgroup analysis examined differences in agreement by rater experience level (senior versus junior raters).

Key Findings

A total of 105 adult cardiac arrest resuscitations were recorded, with 80 paired assessments available for analysis. The mean global TEAM ratings were nearly identical between live and video assessments (8.25 versus 8.22; mean paired difference -0.03, 95% CI -0.25 to 0.20), suggesting no systematic bias at the group level.

However, total teamwork scores were modestly lower on video review compared to live assessment (mean paired difference -1.00, 95% CI -1.77 to -0.23), indicating a slight tendency to score less favorably when reviewing recordings retrospectively.

Event-level agreement was limited, with ICCs of 0.21 (95% CI 0.00 to 0.41) for total TEAM scores and 0.16 (95% CI -0.06 to 0.37) for global ratings, reflecting poor-to-fair reproducibility when examining individual cardiac arrest events. Bland-Altman analysis revealed wide limits of agreement, with video ratings averaging 1.24 points lower than live ratings and limits stretching from -12.28 to 9.80 points, highlighting substantial within-case variability.

Importantly, senior raters exhibited higher inter-method agreement than junior raters, suggesting experience improves consistency in scoring between live and video modalities.

Expert Commentary

These findings contribute to an emerging understanding that while video review can reliably capture group-level quality of teamwork during resuscitation, subtle nuances and contextual factors may elude retrospective rating, leading to discrepancies in event-level evaluations. The modest reduction in scores on video review could stem from increased critical scrutiny or loss of immersive situational awareness familiar to live raters.

The notably poor to fair ICC values underscore the need for cautious interpretation of video-based assessments when used for high-stakes decisions such as individual clinician performance evaluation or credentialing. As demonstrated, rater expertise materially influences the reliability of such assessments, reinforcing recommendations for employing experienced evaluators trained in TEAM methodology and potential multi-rater approaches or structured adjudication.

Limitations include a single-center study design and the use of only one rater per event, which may limit generalizability and precision. Future research might focus on enhancing video review protocols, incorporating multi-angle recordings, and developing rater training programs to mitigate variability.

Conclusion

Delayed video-based TEAM assessment of nontechnical skills during adult cardiac arrest resuscitation events yields similar average scores as immediate live assessment but shows limited agreement at the level of individual resuscitations. Consistency improves with senior rater experience. Video review remains a valuable tool for team debriefing and quality monitoring; however, for precise evaluation of specific resuscitations—especially for professional appraisal—experienced raters and structured safeguards are essential to ensure reliable interpretation.

Funding and ClinicalTrials.gov

The study was supported by institutional grant funding; no clinical trial registration was reported.

References

  1. Lien CJ, Yu PH, Huang SK, Chang JH, Pei-Chuan Huang E. Live Versus Delayed Video-Based TEAM Assessment During Real-World Cardiac Arrest: Agreement and Exploratory Differences by Rater Experience. Annals of Emergency Medicine. 2026 Nov 6; PMID: 42998077.
  2. Cooper S, Wakelam A. Teamwork and non-technical skills in resuscitation. Postgraduate Medical Journal. 1999;75(886):405-409.
  3. O’Connell KJ, O’Connor P, Kerr KM, Smith CM. Accuracy and reliability of video-assisted evaluation of resuscitation teams: a systematic review. Resuscitation. 2018;126:40-46.
  4. Cooper S, Cant R, Kelly M. Non-technical skills in resuscitation. Resuscitation. 2010;81(6):700-706.

Specialty calculators

Browse calculators

For clinical reference; use professional judgment.

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply