Examining Background Knowledge Bias in a High Stake General Academic Language Test: a Differential Item Functioning Analysis

The present study employed a Rasch model to investigate the presence of Differential Item Functioning (DIF) in the performances of participants on a high stakes language proficiency test in a foreign language learning context. A general academic English test which consists of three separate sections, i. e. structure, vocabulary, and reading comprehension, was used in this study. The participants (N=۵۲۳۶), screened from a population of ۷۳۵۵ examinees, belonged to two general academic backgrounds, i. e. Humanities (N=۳۵۸۵) and Science and Technology (N=۱۶۵۱). The DIF analysis showed that ۶۸ percent of the items were functioning differentially for the two groups. The distribution of DIF items in the three subtests was not equal. As the Science and Technology group outperformed the Humanities students on both the total test and an item composite constructed out of only neutral items which displayed no DIF, it may be concluded that the presence of a large number of items on the test has not led to bias against any group. While the majority of the DIF items in the grammar part were in favor of the Science and Technology students, the DIF items in the reading and vocabulary sections mostly favored the Humanities group. While the subtests may be biased against a particular group, the distribution of DIF items in the total test may be such that the total test is not biased. Hence, it is suggested that both the total test and the subtests be examined in bias analysis.

Keywords:

Diffrential Item Functioning , Bias , High-Stake Tests , Background Knowledge