select search filters
briefings
roundups & rapid reactions
before the headlines
Fiona fox's blog
The Office of Qualifications and Examinations Regulation (Ofqual) have published* an interim report detailing how grades have been awarded for secondary education qualifications in summer 2020 after exams were not sat.
Prof George Constantinides, Professor of Digital Computation & Royal Academy of Engineering / Imagination Technologies Research Chair, Imperial College London, said:
“The algorithm used a problematic methodology to adjust for prior cohort attainment, likely to lead to big surprises and disadvantaging centres which do well in narrowing gaps in prior attainment by the time of GCSE / A-Level outcomes. Some of the testing is questionable, in particular the substitution of mark rank orders for teacher-supplied rank orders for 2019 results; this leads to over-estimates of the accuracy of the algorithm. It is also unfair, in my view, to use rank-order based grades for some students but not for others in small cohorts. Fundamentally, quantification of uncertainty is extremely important in any statistical method, and I do not believe that awarding any single grade in this manner adequately reflects the uncertainty involved. At this late stage, the only option available is to withdraw the algorithm and award unmodified teacher-assessed grades. In the future, we should be building more robust assessment systems and tackling the fundamental inequalities at the root of centre-level variation in outcomes.”
Prof Constantinides has also written a blog on this topic: https://constantinides.net/2020/08/15/a-levels-and-gcses-in-2020/
Dr Tom Fincham Haines, UKRI Centre for Doctoral Training in Accountable, Responsible and Transparent AI, University of Bath, said:
“This was a complicated problem, for which they made several fundamental technical mistakes in the design and implementation of the algorithm.
“Of greatest social impact is clearly that small classes were omitted from the algorithm, where they reverted to the teachers grades which tend to be a little optimistic. Put simply, students taught in small schools did well. Secondly, when testing the algorithm they asked it to predict the 2019 results, but gave it the same 2019 results, so the algorithm could cheat. This mean that instead of choosing the approach that was best at predicting students grades, they choose the algorithm that was best at cheating! Additionally, the maths was poorly thought through, such that even great schools that have consistently got C and better grades were forced to have at least one student with a U.
“The algorithm moderated students’ predicted grades in terms of their school’s performance rather than their individual previous performances, meaning the students themselves had very little input into their grades.
“The obsession with thinking in terms of schools rather than individual students needs to change, especially as this appears to disproportionately benefit private schools.
“This is a fundamental example of why AI ethics matters. We need a change in mindset on how we use AI to solve complex problems.
“We wouldn’t build a bridge without running simulations of its design, building models, validating the engineering, checking the safety and implementation in relation to building regulations.
“In a similar way we should be checking and validating the safety of AI systems before we implement them for potentially life-changing applications.
“Ideally the algorithm should have been published earlier so that it could be scrutinised and checked by external experts.
“The very idea of fairly guessing a student’s grade is problematic, and only acceptable if you forget we are discussing real people, with individuality, dreams and purpose. They would have certainly been happier sitting their exams, whatever it took to do so safely.”
Dr Haines has also written a blog on this topic: http://thaines.com/. And created an interactive visualisation showing how the algorithm worked out grades: http://thaines.com/content/ALevels/vis/
Declared interests
None received.