Dinosaur data: the problem with long-term statistics

If, as Harold Wilson famously noted, a week is a long time in politics, then 50 years is surely several aeons.
 
As you might expect therefore, the claims made by politicians that make reference to such lengthy periods are often fraught with danger.
 
One such example was made yesterday by Health Secretary Andrew Lansley, who was quoted in a Departmental press release as saying that “over fifty years, we have halved the proportion of adults who smoked.”
 
On one level, this is a fairly uncontroversial claim. The rapid decline in the numbers of people lighting up since a link to cancer was established in the 1950s has been well documented in this country as well as others.
 
The best available data does indeed show the sort of trend identified by Mr Lansley. According to Cancer Research UK, in 2008 (the most recent year for which data is available) 22 per cent of men and 21 per cent of women smoked. In 1958 the equivalent proportions were 58 and 37 per cent respectively.
 
Amongst all adults therefore the proportion of smokers has fallen from 47.5 per cent to 21 per cent over the last half century, an even greater drop than that which was highlighted by the Health Secretary.
 
But on another level there are a number of statistical nuances that such a broad assertion can never account for.
 
A closer look at the data reveals a number of difficulties.
 
For example, the Department for Health currently uses the Office for National Statistic’s (ONS) General Lifestyle Survey as the source for its smoking statistics.
 
However this survey was first carried out in 1971, and therefore cannot support an analysis covering the past half century.
 
As a spokesperson for the Department explained to Full Fact, “the General Lifestyle Survey only goes back 30 years, so we relied on an individual at the NHS Information Centre to provide figures for the preceding years.”
 
Given that the methodology for calculating this earlier period is not publicly available, it is difficult to judge how legitimate it is to compare its results to those of the ONS data.
 
Inevitably when working with such a long timeframe, there are also a number of important methodological changes that have been made that could impinge upon the accuracy of the information.
 
Most notably, weighting was only applied to the General Lifestyle Survey in 1998, meaning that older data didn’t account for any divergence between the backgrounds of its respondents and the demographic make-up of the country as a whole.
 
As the methodological notes to the first survey show, the sample used was self-selecting, in that it relied upon people being available and willing to respond to questions. A comparison of the weighted and unweighted data for the 1998 survey shows only small differences, but these could be important over long periods of time.
 
As the Industry watchdog the Market Research Society has previously told Full Fact, unweighted surveys can greatly undermine the accuracy of the results returned.
 
So whilst there is a strong case for drawing similar conclusions to Mr Lansley, we would always urge caution when such findings rely on disparate and incomplete data.

 
 

Our work depends on your donations. Click donate to make a one-off donation.

Full Fact believes in the possibility of accurate, informed, enlightening political debate.

We work with politicians, journalists, experts and the public to stop inaccurate claims being made and spread; to make sure reliable information is available and mistakes are corrected, not vilified.