Detecting Halo Effects across Rubric Criteria in L۲ Writing Assessment: A Many-facet Rasch Analysis

This study applied multi-faceted Rasch measurement to investigate the halo effect in the performance-based assessment of writing across rubric criteria. Five raters who had received specialized training applied a four-criteria rating rubric to analytically rate writing scripts on two argumentative topics. Facets, a Rasch computer program, was utilized to pinpoint the halo effect by analyzing rater and rubric criteria interactions. After examining the appropriateness of the rubric in terms of functionality, the results showed that except for one rater, the raters did not exhibit any sign of the halo effect across rubric criteria. Generally, the severity hierarchies for raters and difficulty levels for rubric criteria suggested that raters’ training and their perceptions of the importance of criteria were associated with their manifestation of the halo effect. Pedagogically, through a detailed facet analysis of interactions between raters and rubric criteria, rater trainers may better realize how to design effective training programs to minimize raters’ variance including the halo effect and improve the overall objectivity of human rating.

Keywords:

Halo effect , Rubric criteria , Rater Variance , Many-facet Rasch measurement (MFRM)