Inter-Rater Reliability of a Suture Assessment Tool
PDF

Keywords

Suture assessment
surgical skills
reliability
family medicine
residency

Categories

How to Cite

1.
Patel K, Lundstrom D, Husted E, Stacey SK. Inter-Rater Reliability of a Suture Assessment Tool. IMJ Translational Med. 2025;2(1). doi:10.5281/zenodo.15698667

Abstract

Introduction: Despite significant advancements in standardizing medical knowledge assessment, evaluating technical skills, such as suturing, has largely depended on subjective faculty evaluations, creating a gap in reliable, objective assessment. Addressing this need, Sundhagen et al. developed a suturing assessment tool, demonstrating promising reliability and validity in standardized settings with medical students. However, its performance remains untested when used by family medicine faculty to evaluate residents in a typical clinical environment. This pilot study aims to assess the inter-rater reliability of the modified Sundhagen tool in evaluating suturing skills among family medicine residents in real-world settings.

Methods: To assess inter-rater reliability, a modified version of the Sundhagen assessment tool, comprising eight yes-or-no questions on suturing skills, was provided to four family medicine faculty members in La Crosse, WI. Faculty members used this tool to evaluate five videos of family medicine residents performing simple sutures on chicken breast incisions, captured during routine educational activities without specific training on tool use. The videos, obtained from a prior residency study with participant consent, displayed only the residents’ hands and suturing materials. Task completion scores were tallied, and inter-rater reliability was measured using ICC, with analysis by question.

Results: The ICC score was 0 (95% confidence interval -0.18 – 0.61), suggesting poor inter-rater reliability.

Conclusion: This pilot study did not demonstrate strong inter-rater reliability for the modified Sundhagen tool when applied to family medicine residents, highlighting the need for further refinement and assessment of additional validity components. As an essential preliminary step, these findings lay the groundwork for a larger, adequately powered trial that can more rigorously evaluate the tool's performance across a broader set of metrics, enhancing its applicability and reliability in family medicine residency training.

PDF

References

Faulkner H, Regehr G, Martin J, Reznick R. Validation of an objective structured assessment of technical skill for surgical residents. Acad Med. 1996;71(12):1363-1365. doi:10.1097/00001888-199612000-00023

Swift SE, Carter JF. Institution and validation of an observed structured assessment of technical skills (OSATS) for obstetrics and gynecology residents and faculty. Am J Obstet Gynecol. 2006;195(2):617-623. doi:10.1016/j.ajog.2006.05.032

Anderson DD, Long S, Thomas GW, Putnam MD, Bechtold JE, Karam MD. Objective Structured Assessments of Technical Skills (OSATS) Does Not Assess the Quality of the Surgical Result Effectively. Clin Orthop Relat Res. 2016;474(4):874-881. doi:10.1007/s11999-015-4603-4

Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273-278. doi:10.1046/j.1365-2168.1997.02502.x

Kramp KH, van Det MJ, Hoff C, Lamme B, Veeger NJ, Pierie JP. Validity and reliability of global operative assessment of laparoscopic skills (GOALS) in novice trainees performing a laparoscopic cholecystectomy. J Surg Educ. 2015;72(2):351-358. doi:10.1016/j.jsurg.2014.08.006

Asif H, McInnis C, Dang F, et al. Objective Structured Assessment of technical skill (OSATS) in the Surgical Skills and Technology Elective Program (SSTEP): Comparison of peer and expert raters. Am J Surg. 2022;223(2):276-279. doi:10.1016/j.amjsurg.2021.03.064

Sundhagen HP, Almeland SK, Hansson E. Development and validation of a new assessment tool for suturing skills in medical students. Eur J Plast Surg. 2018;41(2):207-216. doi:10.1007/s00238-017-1378-8

Stacey SK, Boswell CL, Cowan KK, et al. Remote Synchronous Laceration Repair Instruction With Summary Feedback. PRiMER. 2023;7:19. Published 2023 Jun 29. doi:10.22454/PRiMER.2023.863182

Brisson BA, Dobberstein R, Monteith G, Jones-Bitton A. Excellent Agreement of In-Person Scoring versus Scoring of Digitally Recorded Simulated Suture Skills Examination. J Vet Med Educ. Published online July 7, 2022. doi:10.3138/jvme-2021-0164

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Copyright (c) 2025 Khyati Patel, Dillon Lundstrom, Elizabeth Husted, Stephen K Stacey (Author)