Constructing and Validating a Q-matrix for Cognitive Diagnostic Analysis of the Listening Comprehension Section of the IELTS

A critical component of cognitive diagnostic models (CDMs) is a Q-matrix that stipulates associations between items of a test and their required attributes. The present study aims to develop and empirically validate a Q-matrix for the listening comprehension section of the International English Language Testing System (IELTS). To this end, a listening comprehension test of the IELTS was administered to ۸۲۰ Iranian test takers. According to theories, taxonomies, and models of second/foreign language (L۲) listening comprehension, previous studies on the utility of CDMs to L۲ listening comprehension, detailed content analysis of the test items, and consultation with several content experts, an initial Q-matrix was first developed. Through the technique suggested by de la Torre and Chiu (۲۰۱۶), along with checking heatmap plots and mesa plots using the GDINA package in R, the Q-matrix was then empirically validated. Generally, six attributes were extracted for the listening section, namely, (۱) Linguistic knowledge (LKA), (۲) understanding prosodic patterns (UPP), (۳) ability to understand and make paraphrases (PAR), (۴) ability to understand specific factual information such as names, numbers, and so forth (UFI), (۵) ability to understand explicit information (UEI), and (۶) ability to make inference (INF). Finally, the results of the fit of the GDINA model to the data, at both item and test levels, indicated the adequate model-data fit and the plausibility of the Q-matrix. The implications of the study were also discussed.

Keywords:

attributes , Cognitive Diagnostic Models (CDMs) , IELTS , listening comprehension , Q-matrix