Statistics watchdog 'concerned' by Department for Education claim

Andrew Dilnot, Chair of the UK Statistics Authority, has today expressed 'concern' about claims made by the Department for Education (DfE) on the UK's supposed slump in international school achievement league tables.

Nearly two years ago, Full Fact took issue with a claim that was proving popular with Education Secretary Michael Gove, who was among those to argue that "the UK [has] slipped from eighth to 28th in maths, from seventh to 25th in reading and from fourth to 16th in science" between 2000 and 2009 in the OECD's Programme for International Student Assessment (PISA). 

In September the same comparison was made on the BBC by the Chief Inspector of Education, Sir Michael Wilshaw, prompting former Schools Minister David Miliband to ask the UK Statistics Authority to comment on the claim.

As we pointed out in 2010, the OECD itself had warned against making these comparisons, saying that "for the United Kingdom it is only possible to compare 2006 and 2009 data." However the Department stuck to its guns when we contacted them, pointing towards research it commissioned to address some of the inadequacies in the earlier data.

Other academics told Full Fact that this didn't fully address the problems with the sort of comparisons being made by the Education Secretary. In search of some clarity, we asked the UK Statistics Authority to intervene.

Mr Dilnot's letter to Mr Miliband suggests that the limitations associated the comparison between the 2000 and 2009 data were not made sufficiently clear by the DfE. He writes:

"I was concerned to review the Department for Education's press release of 7 December 2010 in which headline results for England from the PISA study, alongside relative international rankings, were not accompanied by detailed advice or caveats to help the reader in making comparisons over time, nor were the statistical implications of an increase in the number of reporting countries in later PISA studies noted."

While the PISA data is not an 'official statistic', and so falls outside the Authority's remit, Mr Dilnot does argue that the same standards should apply when the DfE discusses its findings:

"it would have been helpful had the Department's December 2010 press release attempted to meet standards similar to those required of departmental statistical publications."

Mr Dilnot also stresses the importance of seeking guidance from Department statisticians when preparing number-heavy releases, and to publish any issues that might be drawn out. It would certainly be concerning if this advice wasn't followed in this instance.

This isn't to say that it is necessarily wrong to draw comparisons between the UK's PISA ranking at the turn of the millennium to the place it found itself in 2009, but that when this trend is highlighted we should also have our attention drawn to the caveats that tell us what the data means. As Mr Dilnot puts it:

"These uncertainties and weaknesses are not just a technical footnote; they are themselves an important part of the evidence, and affect interpretation and meaning. League tables and the presentation of international rankings can be statistically problematic, and require clear and careful commentary alongside them."

While the Statistics Authority has been looking into the issue, we have noticed that the DfE hasn't been as vocal in repeating this claim. We hope the Department has therefore already taken on this advice, and that the public will get a better understanding of the level of attainment British students are reaching when compared to other developed countries.

As head of Ofsted, Sir Michael Wilshaw is independent of the Department for Education and therefore presumably did not get the memo. We would hope that the Authority having its made its point will ensure that this comparison is kept in its proper context in future and Full Fact will continue to keep an eye out for slips.