174 / 2018-08-11 20:50:40

A Study on Cognitive Diagnostic Assessment of EFL Writing Ability

EFL Writing Assessment; Cognitive Diagnostic Assessment; Rating checklist; Q-matrix Model; Multifaceted Rasch Model

全体主题 > 面向职场的语言测试与评价

This study aims to construct a cognitive diagnostic assessment model for Chinese College English writing (CCEW) by developing and validating an empirically-derived, descriptor-based（EDD）writing checklist, so as to produce fine-grained, detailed cognitive diagnostic reports. The study was conducted in the two-phases, with the establishment of EDD rating checklist first, followed by Q-matrix model construction, and then model-fit analysis and generation of diagnostic feedback report. Through mixed methods such as Think-aloud protocal (TAP) in extracting descriptors for the rating checklist, Multifaceted Rasch Analysis for its validation, expert-coding for constructing Q-Matrix and Reduced RUM (CDA model) for generating the diagnostic report. The results show that 1) 30 descriptors extracted from raters’ TAP corpus, are consistent with EFL writing theories; 2) the descriptor-based rating checklist is proved to be reliable, consistent, and can distinguish rating performance across raters; 3) five attributes identified—Content, Organization, Language Use, Vocabulary and Mechanics—are relevant to the construct of the testing domain. 4) the Q-Matrix coded by the experts are proved to be reliable and valid through model-fit analysis; 5) the finally validated Q-matrix as the Cognitive Diagnostic Writing Model is able to classify 91.39% of the test takers into eight types of skill mastery patterns and 83.31% of candidates can be successfully classified into five groups of skill mastery patterns (11111, 00000, 01111, 01001, and 11001), which indicates a strong power of skill mastery profiling. As a result, students’ score report as mastery profile is produced with detailed descriptions on weakness and strengths of the attribute mastery so that the targeted instructional remedial advice could be given against their weaknesses. The study significantly facilitates the development of personalized assessment and "tailored" instruction.