Evaluating the Interrater Reliability for the Intern Selection Process Done at Providence Medical Group Psychology Training Program
Files
Publication Date
4-29-2026
Keywords
oregon, pmg, pmg psych gme
Disciplines
Medical Education
Abstract
Interrater reliability (IRR) is an important component of evaluation processes that involve subjective judgments, as it reflects the degree of consistency among multiple raters. In training and hiring contexts, establishing adequate IRR helps ensure that applicant evaluations are fair, systematic, and minimally influenced by individual bias. The present program evaluation examines the interrater reliability of the intern selection process at the Providence Medical Group Psychology Training Program. Specifically, this project evaluates the consistency of raters’ scores assigned to internship applicants using the program’s standardized application rating forms. Numerical ratings from multiple reviewers were collected for each applicant and analyzed using Cronbach’s alpha to assess the level of agreement among raters. The analysis focuses solely on the application review stage of the selection process; interview ratings were excluded to minimize potential bias introduced through interpersonal interactions. By examining the internal consistency of application ratings, this evaluation aims to determine whether the current scoring procedures yield reliable assessments of applicants. Findings from this project may inform future refinements to the internship selection process, including clarification of rating criteria and improvements to reviewer training, ultimately supporting a more equitable and consistent evaluation system.
Specialty/Research Institute
Graduate Medical Education
Specialty/Research Institute
Internal Medicine
