Skip to main content

Research Repository

Advanced Search

Standard setting in Australian medical schools

Ward, H.; Chiavaroli, N.; Fraser, J.; Mansfield, K.; Starmer, D.; Surmon, L.; Veysey, M.; O'Mara, D.

Authors

H. Ward

N. Chiavaroli

J. Fraser

K. Mansfield

D. Starmer

L. Surmon

M. Veysey

D. O'Mara



Abstract

© 2018 The Author(s). Background: Standard setting of assessment is critical in quality assurance of medical programs. The aims of this study were to identify and compare the impact of methods used to establish the passing standard by the 13 medical schools who participated in the 2014 Australian Medical Schools Assessment Collaboration (AMSAC). Methods: A survey was conducted to identify the standard setting procedures used by participating schools. Schools standard setting data was collated for the 49 multiple choice items used for benchmarking by AMSAC in 2014. Analyses were conducted for nine schools by their method of standard setting and key characteristics of 28 panel members from four schools. Results: Substantial differences were identified between AMSAC schools that participated in the study, in both the standard setting methods and how particular techniques were implemented. The correlation between the item standard settings data by school ranged from - 0.116 to 0.632. A trend was identified for panel members to underestimate the difficulty level of hard items and overestimate the difficulty level of easy items for all methods. The median derived cut-score standard across schools was 55% for the 49 benchmarking questions. Although, no significant differences were found according to panel member standard setting experience or clinicians versus scientists, panel members with a high curriculum engagement generally had significantly lower expectations of borderline candidates (p = 0.044). Conclusion: This study used a robust assessment framework to demonstrate that several standard setting techniques are used by Australian medical schools, which in some cases use different techniques for different stages of their program. The implementation of the most common method, the Modified Angoff standard setting approach was found to vary markedly. The method of standard setting used had an impact on the distribution of expected minimally competent student performance by item and overall, with the passing standard varying by up to 10%. This difference can be attributed to the method of standard setting because the ASMSAC items have been shown over time to have consistent performance levels reflecting similar cohort ability. There is a need for more consistency in the method of standard setting used by medical schools in Australia.

Citation

Ward, H., Chiavaroli, N., Fraser, J., Mansfield, K., Starmer, D., Surmon, L., …O'Mara, D. (2018). Standard setting in Australian medical schools. BMC Medical Education, 18(1), Article 80. https://doi.org/10.1186/s12909-018-1190-6

Journal Article Type Article
Acceptance Date Apr 12, 2018
Online Publication Date Apr 23, 2018
Publication Date Apr 23, 2018
Deposit Date Aug 13, 2019
Publicly Available Date Mar 29, 2024
Print ISSN 1472-6920
Publisher BioMed Central
Peer Reviewed Peer Reviewed
Volume 18
Issue 1
Article Number 80
DOI https://doi.org/10.1186/s12909-018-1190-6
Keywords Standard setting; Assessment; Preclinical teaching; Medical education
Public URL https://hull-repository.worktribe.com/output/1932478
Publisher URL https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-018-1190-6

Files

Article (893 Kb)
PDF

Copyright Statement
© The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and
reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to
the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver
(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.





You might also like



Downloadable Citations