The media management of benefit stats

Update: Since this piece was published, the head of the UK Statistics Authority, Andrew Dilnot has written to the Work and Pensions Secretary about his use of the benefit cap statistics.

This week people in four London boroughs have seen a cap imposed on their state benefits. With a few exceptions, there will be a £500 per week limit for a couple and a £350 per week limit for those living on their own. This is the initial trial phase of a new scheme that aims to bring benefit payments into line with wages. The Government wants to ensure that those living on out-of-work benefits do not receive more than someone employed on an average income.

Despite it being early days, the Department for Work and Pensions (DWP) is already claiming that its policy has been successful. Last week it released a series of numbers suggesting that fewer households than expected would be affected by the benefit cap and that some of those who would potentially have been subject to it are now back in work.

However, the UK Statistics Authority, which monitors the use of official statistics, is now reviewing how the DWP handled these statistics. The way the DWP released the figures has certainly generated controversy. Here's how it happened.

Keeping it ad-hoc

On Friday, ahead of the introduction of the benefit cap on Monday, the DWP Press Office issued the following tweets, two minutes apart: 

And, for the uninitiated, here are the tweets translated and in a more readable format:

1. Jobcentre Plus (JCP) has helped 8,000 claimants who would potentially come under the benefit cap into work.

2. Figures show a 25% drop in the number of people affected by the household benefit cap, from 56,000 to 40,000.

To anybody monitoring the DWP's Twitter feed, the figures in these tweets might reasonably appear to be linked. Both statistics refer to those affected by the new benefit cap. At the same time, it looks like the DWP press office appears to have been briefing the media on both sets of numbers (more on that later).

In fact, there is no direct relationship between the numbers. These are 'ad hoc' statistics - that is, each number is from a statistical analysis that isn't part of one of the Department's standard publications. Ad hoc statistics are often produced to assist a Department in briefing the media, or in response to a specific question that it has been asked.

The figures in each of the tweets appear in separate statistical notes published by the Department. This much suggests that the DWP's statisticians are not linking the numbers. More importantly, each statistical release has been produced using different sets of data. The figure of 56,000 to 40,000 households (the 25% drop in those affected by the benefits cap) is based on data from the benefit payment systems. The other number - the 8,000 increase in the number of claimants in work - has been generated using labour market survey data.  

There's no departmental press release that shows that the DWP is officially linking the numbers. However, to the average reader it might look like the statistics are part of the same story. It would be perfectly reasonable for anybody monitoring the DWP's tweets (or the subsequent press coverage) to understand that of the 16,000 people who are no longer subject to the benefit cap, half of them (8,000) have found employment.

And, as we can see, this is how it was reported. Two tweets became one sentence:

Here's, for instance, how the Guardian framed the numbers:

"The government last week claimed the number of people expected to be hit by the cap had fallen from 56,000 to 40,000, with 8,000 claimants finding work through JobCentre Plus."

Exactly the same sentence appeared in the Independent.

This may have something to do with the fact that the Press Association, which supplies many national newspapers with wire reports, had implied a link between the figures.

Transparency first?

And there's another problem. We would expect to find both statistical releases in the same place - the ad hoc statistics section of the DWP's website. But only one of the releases (for households) is here; the other (for claimants) is buried in a different area of the website. At the time of writing, we can find it either in the Local Authority section or in the Advisers and Intermediaries part of dwp.gov.uk. 

When we asked the DWP to clarify why you can't find both statistical releases on the same page, a spokesperson declined to answer the question.

Although the DWP was the first government department to publish ad hoc statistics, it is bizarre that on this occasion it has not followed its usual procedures. 

The reason why due process is important is because it helps the media - and the public - to interpret statistics that are often complicated and frequently caveated.

What are the facts? 

On the one hand, we're being told about a decrease in the number of households subject to the benefit cap. On the other, we're being presented with a different set of numbers: the increase in the number of people (claimants) being helped into employment.

The numbers are, quite simply, not comparable. After all, a household is often comprised of several people. 

To treat these figures as part of the same whole is like comparing the number of horses slaughtered in England with the kilograms of ready-made lasagne eaten in the UK each year. Both numbers are relevant in a discussion about horse meat ending up in 'beef' ready meals, but it would be wrong to state that one is a direct result of the other. 

Each number is in itself 'respectable': it tells a story. But the DWP mangled the message.

Media (mis)management?

For a start, the DWP confused 'households' with 'people' in its tweet. This is nothing short of an invitation to compare the figures (whether or not that invitation was issued accidentally).

While we don't know how journalists were briefed behind the scenes, they have repeated the DWP's mistake. However, the DWP ought to explain why it presented its statistics in this way. 

As we've noted, the DWP has previously been criticised for the way that it has handled the release of benefits statistics. In the case of the benefit cap statistics, we asked the DWP whether it was suggesting that there was a link between a set of incomparable numbers. We received the following statement as a response:

"Jobcentre Plus data clearly show that 8,000 people who would have been affected by the benefit cap have moved into work and 25,000 are accepting help to get a job. 

We have followed the correct procedures for publishing this data and it is available for anyone to study. Claims to the contrary are utterly unfounded."

It's hardly surprising that there has been some confusion over what the statistics mean. A spokesperson from the UK Statistics Authority (UKSA) has said, "The Authority is reviewing the matter and will respond publicly in due course." 

At a time when the Government's policies on benefits are making daily headlines, it's more important than ever that the debate is an informed one.

If the DWP is not clear in its presentation of this type of statistics, then there is a risk that the public's confidence in the political process will be further undermined. Sir Michael Scholar, former chair of the UKSA, has previously emphasised how important it is that the DWP publishes ad hoc statistics "in a transparent way" because "this is central to retaining trust in official figures".

It's now up to the statistics watchdog to take a view on these latest concerns about DWP's media management.