The validity, reliability and responsiveness of procedure based assessment in simulated vascular procedures

Green, Lucy Anne

Abstract

Introduction

Procedure based assessment (PBA) has been shown to be valid and reliable in the workplace however whether this translates to the simulation setting has not yet been demonstrated. Therefore the aim of this thesis is to demonstrate that PBA is a valid, reliable and responsive assessment tool in simulated vascular procedures.

Methods

Three experiments based on simulated vascular operations were designed to explore the validity, reliability and responsiveness of PBA utilising 3 commonly performed vascular procedures. The global and task specific checklist (GTSC) and global summary score (GSS) of a modified PBA were analysed separately. Validity was determined by correlating performance with prior operative experience (number of operations previously observed and performed) and stage in surgical training. Reliability and responsiveness was determined by use of multiple raters and assessing
change in performance over time.

Results

The modified PBA was found to be a valid assessment method based on number of operations previously performed (r=0.446 p=0.029 for the GTSC and r=0.553 p= 0.005 for the GSS) but not for operations previously observed. Only the PBA GTSC was valid for stage of surgical training (r= 0.588 p=0.002). The modified PBA demonstrated good inter-rater reliability (r= 0.665 p= 0.005 for the GTSC and r= 0.843 p> 0.001 for the GSS) during simulated vascular procedures. Intra-rater reliability was not demonstrated. The PBA GSS was found to be responsive to improved performance (WSR p< 0.001) but the PBA GTSC was not (WSR p = 0.104).

Conclusion

The modified PBA is a valid assessment of surgical skill when correlated with previous operative performance. Observation alone appears to contribute little to assessment outcomes. Performance is index specific and not fully dependant on training level. PBA has only partial reliability in simulated vascular procedure due to the lack of intra-rater reliability. PBA was responsive to practice effect suggesting it could be useful to monitor trainee performance in simulation. PBA potentially has a role in simulation assessment but did not demonstrate sufficient reliability for high stakes examination.

Citation

Green, L. A. (2014). The validity, reliability and responsiveness of procedure based assessment in simulated vascular procedures. (Thesis). Hull York Medical School, the University of Hull and the University of York. Retrieved from https://hull-repository.worktribe.com/output/4216921

• Thesis Type: Thesis
• Deposit Date: Sep 2, 2015
• Publicly Available Date: Feb 23, 2023
• Keywords: Medicine
• Public URL: https://hull-repository.worktribe.com/output/4216921
• Additional Information: Hull York Medical School, The University of Hull and the University of York
• Award Date: Sep 1, 2014