Setting a common standard in clinical skills assessment: The experience of the California Consortium for the Assessment of Clinical Competence

Objective or purpose of innovation: To identify common clinical skills competency thresholds across schools by centralizing standard setting for a multi-institutional assessment.

Background and/or theoretical framework and importance to the field: The California Consortium for the Assessment of Clinical Competence (CCACC) comprises 10 medical schools that administer a common multi-station clinical skills assessment (CPX). Previously, each institution determined their own, largely norm-referenced passing thresholds for the examination. With the elimination of USMLE Step 2 CS, there is a recognized need for robust clinical skills assessment beyond the individual institutional level. A collaboratively developed, multi-institutional examination with passing thresholds established via a rigorous process offers greater validity evidence for summative decisions made based on its results. Accordingly, the CCACC undertook centralized, criterion-based standard setting for the CPX.

Design: Passing thresholds for the six core CPX cases were determined via two methods: modified Angoff, using expert raters from multiple institutions, and borderline regression, using global encounter ratings assigned by standardized patients. Results from the two methods were compared to each other and to institutions’ prior thresholds.

Outcomes: Both methods yielded the same cumulative cut score based on averages across all cases (70%), but exhibited variation between individual cases, suggesting case-specificity. Compared with prior thresholds, some institutions’ pass rates would have been higher using the common criterion-referenced cut score, while others would have been lower.

Innovation’s strengths and limitations: This study demonstrates the feasibility of centralizing standard setting across multiple institutions using two criterion-based methods. Standardized patient ratings may generate similar passing thresholds to those determined by clinicians. Further studies are necessary to determine whether these findings generalize to other case types and how best to apply centralized standards within each institution’s context.

Feasibility and generalizability: The CCACC’s standard setting approaches may be applied across other institutions sharing an assessment, allowing for comparison of learner performance to a common standard. Given the similar results, the choice of method may be determined by resource availability.