A push to introduce automated essay scoring (AES) to mark NAPLAN online writing tests could undermine the value of test results, the Federation of Parents and Citizens Association of NSW (the Federation) president Susie Boyd said.
Subscribe now for unlimited access.
$0/
(min cost $0)
or signup to continue reading
September’s meeting of the federal Education Council gave in-principle support for NAPLAN writing tests to be marked by a computer – but double-checked by a person – in 2018.
But leading US academic Les Perelman’s recent analysis of a 2015 paper by Australian Curriculum, Assessment and Reporting Authority (ACARA) has revealed there are potential flaws in using AES.
According to Perelman’s analysis, AES has not been shown to measure the content of written work, but rather qualities like sentence structure and vocabulary, Ms Boyd said.
Consequently, AES platforms may award high marks to random gibberish, as long as it uses sophisticated vocabulary and sentence structure.
A recent independent survey commissioned by the Federation revealed only seven per cent of respondents believed that a computer system could mirror human marking.
“If a student just takes time to memorise some well-written gibberish, could that improve their NAPLAN scores under AES?” Ms Boyd commented.
“If so, that would completely undermine the value of NAPLAN results, and it’s disturbing that ACARA hasn’t even looked into that possibility.
“Students may not be able to game the system like this during the initial trial period, as ACARA has said automated marking will be double checked by humans.
“However, how will ACARA ensure this doesn’t happen if human markers are removed completely?”
Research from Pearson Education – the company developing AES platforms for ACARA – explicitly states that AES systems were not good at assessing rhetorical voice, the logic of an argument, the extent to which particular concepts are accurately described, or whether specific ideas presented in the essay are well founded, Ms Boyd said.
The developers also conceded the sassessment of creativity, poetry, irony, or other more artistic uses of writing was beyond such systems.
“If you look at the NAPLAN marking criteria for writing, it includes qualities like persuasive devices and the ability to engage the reader,” Ms Boyd said.
“Given Pearson Education’s own conclusions, we shouldn’t trust the ability of AES systems to implement NAPLAN’s marking criteria.
“We call on ACARA to focus on what’s in the best interests of our children and abandon its plans to introduce AES to assess our children’s creative work.”
