Experienced but detached from reality: Theorizing and operationalizing the relationship between experience and rater effects

Abstract

It is often argued that, to achieve a high quality of rating in writing assessment, it is important to nurture a comprehensive measurement ecosystem involving experienced raters and appropriate rating scales. But what is ‘experience’, and how do different kinds of experience affect the severity and the reliability of rating? We collected data from a high-stakes English writing examination to investigate how 18 raters with temporally and qualitatively different experiences interpret and use a rating scale. We analyzed the data using mixed methods, including content analysis, Rasch models and Graph theory showing that qualitatively different experiences affect severity in different ways. Also, irrespective of past experience, raters who disengaged from their Community of Practice (CoP) were more likely to yield unreliable ratings. These results have important methodological implications in how researchers theorize and operationalize ‘rating experience’. Our findings highlight the importance of active and uninterrupted engagement in raters’ CoP with implications for both researchers and policy makers.