Free schools' performance: statistics watchdog says experts are unresolved on a fair comparison
"The question of how to compare free school inspection outcomes with other schools in a fair way appears to remain unresolved by the experts."
Those are the words of the UK Statistics Authority (UKSA) in response to a letter we sent to its Head of Assessment regarding claims by the Department for Education that free schools are twice as likely to be 'outstanding' under the new Ofsted framework.
In June we wrote about a press release issued by the department claiming that:
"Sixty-nine per cent of open free schools have been rated 'good' or 'outstanding' under the new Ofsted framework, compared to 64% of all schools. Free schools are twice as likely to be 'outstanding', with 21% of open free schools rated 'outstanding' compared to 10% of all schools under the same framework."
We had repeatedly asked the department's press office for the details behind the claim - including the time period in which the inspections were counted, and how many free schools and other schools were included in their comparison - but without reply. So we asked the UK Statistics Authority to look into it.
In our letter we also raised concerns about the problems with free school data, since the claim not only relies on a comparison of a very small number of schools to a very large number of schools, but Ofsted had previously told us that some of the free schools may not have older pupils yet (if they've started filling up from Year 7 in their first year for example).
Arithmetically correct but numbers are small
Ofsted and the Department for Education provided the UKSA with the statistics behind the claim, which confirm that it was based on inspections of 62 free schools, of which 13 were judged as outstanding (21%) as at mid June 2014. The Department compared this to inspections of all schools since 1st September 2012 (when the most recent school inspection framework was introduced), finding 10% of all schools were judged outstanding.
The data for the 10% is slightly less clear from the letter: Ofsted told the UKSA that 9% of all schools (1,044 of 11,932) were judged outstanding between 1st September 2012 (when the most recent school inspection framework was introduced) and 31st March 2014. The DfE's analysis refers to more recent data, but it's not clear from the letter how many schools the 10% corresponds to. We've got in touch with the UKSA to check.
The UKSA said that on this basis "It is arithmetically correct to say that the proportion of free schools judged as outstanding is higher than the proportion of all schools so judged". But it notes that:
"However, such comparisons are by no means straightforward. First, the number of free schools inspected is small, and the number judged as outstanding is very small. Across all categories, including schools judged as inadequate, there are differences in both directions between free schools and other schools. Second, there is a likely ('downward') bias in the judgements for maintained schools as presented in the tables, because those previously judged to be outstanding are less likely to be inspected in any given period".
Further analysis forthcoming
We are pleased to see that the Head of Profession for statistics at the Department for Education confirmed to the UKSA that the department plans to imminently publish a piece of analysis that will shed further light on the questions we raised. The UKSA also confirmed that it will discuss with the Head of Profession the importance of making available the full details of statistics of high public profile at the time of release.
Image credits: http://bit.ly/1qGKgjP