171 / 2018-08-11 09:58:28

The effect of task differences on rater cognition in an EFL speaking test

task effect,rater cognition,EFL speaking test

全体主题 > 与语言测试与评价相关的其他议题

While most comparative studies on task effect are based on scores, we believe that more insight can be obtained from comparing raters’ cognitive activities in different tasks. The similarities and differences that result from such comparisons will help us lay bare the task effect on the rating of the common components of the target construct represented by different tasks and the unique components of the target construct represented by individual tasks, and construct-irrelevant components associated with individual tasks.
To explore the value of this cognitive approach to task effect, we explored the different judgmental processes of nine raters when rating two speaking tasks in the Test for English Majors, Band 4, Oral Test (TEM4-Oral), a monologic narrative of personal experience and a dialogic argumentation over a social issue. All the raters were required to justify their ratings in an immediate retrospective. Then they were interviewed to reflect on their rating processes and interpretation and use of the official rubric. The verbal protocols were digitally recorded, transcribed verbatim, and coded in NVivo 11 in the bottom-up approach. The resulting themes were then compared to the official rubric.
The major findings of the data analysis were: a) on the whole, the raters’ justifications agreed strongly with the descriptors in the official rubric; b) however, the raters varied to different extents in evaluating the construct components and construct-irrelevant components. The findings can be interpreted in terms of the interactions between the rater and the rubric, and between the rater and the task (McNamara, 1996). As the findings can be used to guide the design of future tasks, scoring rubrics, and rater-training procedures in similar tests, the study highlights the value of a cognitive perspective in validation and development of language performance assessments with more than one task.