The Impact of Test Length on Raters’ Mental Processes During Scoring Test-Takers’ Writing Performance

Different factors such as the writing genre, writing prompt, and/or test length can influence the raters’ mental processes while scoring writing tests. Accordingly, whether an increase or a decrease in test length has any impact on how raters evaluate test-takers’ writing performance was the motive underlying this research. For this purpose, ۱۲ EFL students who scored between ۵.۵ to ۷.۵ on the writing section of a mock IELTS test were selected based on availability sampling. The participants wrote three argumentative essays (the original, longer, and shorter versions). The three versions from each test-taker were then scored by three raters using IELTS task ۲ writing band descriptors. Meanwhile, the raters provided verbal protocols explaining in detail the reasons underlying their scores to each test-taker’s essay. Then, the verbal protocols were transcribed and content analyzed using Nvivo version ۱۱ to extract the themes mentioned by the raters in scoring each writing test. The results showed that the raters paid more attention to certain factors in the band descriptors and ignored some other factors. However, there was a similar pattern among the raters in scoring the three writing tests. The results did not show any significant differences in the raters’ mental processes while scoring each of the three writing tests. The conclusion was that test length is not a determining factor influencing the mental processes of raters in writing tests. Therefore, raters and test developers do not need to worry about test length influencing the raters’ scoring.

Keywords:

mental processes , rater , test length , verbal protocol , writing test