More confusion from same-sex marriage polls
Last month Full Fact tackled the question of whether the majority of the British public supports same-sex marriage. Since then, a number of new polls have arisen, and readers could be forgiven for being somewhat confused after reading them all:
Telegraph/ICM: "The results show opinion finely balanced, with 45 per cent supporting the move in principle, 36 per cent opposing it, and the rest saying they do not know"
Sunday Times/YouGov: "In a YouGov survey for The Sunday Times today, 43% of those polled said they support gay marriage, while 32% said they supported civil partnerships but opposed gay marriage"
Telegraph/ComRes: "Seven out of 10 people are opposed to the Government's plans to redefine marriage to include same-sex couples, a poll suggests."
Populus: "Latest Gay Marriage stats from last weekend. Two thirds of Brits in favour."
The four recent polls were conducted by ICM, YouGov, ComRes and Populus. Respectively, they seem to indicate that 45, 43, 22 and 65 per cent of the public support gay marriage. Crucially, every poll was reported to show how many people 'support' or 'favour' plans to legalise gay marriage.
As could be expected, different polls have been reported by different organisations. PinkNews.co.uk refer several times to the ICM poll showing 45 per cent support. Meanwhile, the Coalition for Marriage reports the ComRes poll showing 70 per cent in opposition.
But why are the figures so different?
Part of the reason is, of course, in the questions. Each polling organisation used different wording when posing the question. Taking them in turn:
"You might be aware that currently the law allows gay people to enter civil partnerships but they cannot get married. The Prime Minister, David Cameron wishes to legalise gay marriage but some senior members of both the Catholic Church and the Church of England are opposed. Do you support or oppose the move to legalise gay marriage?"
"Since 2005 same-sex couples have been able to enter into civil partnerships. While civil partnerships offer the same legal rights as marriage, same-sex couples are not able to marry. Which of the following best reflects your view?"
1. I would support same-sex marriage
2. I support civil partnerships, but would oppose same-sex marriage
3. I oppose both civil partnerships and same-sex marriage
4. Don't know
"Do you agree or disagree with ... Marriage should continue to be defined as a life-long exclusive commitment between a man and a woman"
"Please say if you agree or disagree with ... Gay couples should have an equal right to get married, not just to have civil partnerships"
The two polls that specifically ask whether the public support "same-sex" or "gay" marriage show similar findings of 45 and 43 per cent support. However, the YouGov poll shows a higher cohort opposed to the move, totalling 47 per cent opposing same-sex marriage compared to 36 per cent for ICM.
YouGov's sample were however given the option to support civil partnerships but oppose same-sex marriage - producing a significant 32 per cent result for this specific option. The ICM poll meanwhile has a much higher 'don't know' result of 19 per cent, while only 10 per cent said this to YouGov.
The ComRes and Populus polls however do not contain specific references to whether the public supports the proposals in general, and instead concentrate on specific points on either side of the argument.
The ComRes poll asks its sample whether it agrees that the definition of marriage should continue as a man-woman commitment - in essence presenting people with a reason to oppose redefinition. Meanwhile the Populus poll mentions the issue of equal rights - in essence presenting people with a reason to support the proposals.
Sure enough, the ComRes poll produces the strongest opposition to the same-sex marriage concept and the weakest support. The Populus poll on the other hand yields the strongest support and weakest opposition of the four polls.
Posing questions of this sort, while valid, can create a risk of a form of 'affirmation bias' - the likelihood that presenting a statement will exaggerate its affirmative support.
This is most glaringly illustrated by a single poll conducted by ComRes in 2009 on European Union membership. Among the questions were two that produced an unusual outcome:
The results of this poll suggest that, in 2009, the majority of people both supported and opposed Britain's membership of the European Union. As polling expert Anthony Wells has pointed out, this created polar opposite results. It is difficult to see how they could be reconciled without considering the question posed.
The ICM and YouGov polls present the most netural statements as the stimulus for soliciting an opinion, while the ComRes and Populus polls present arguments and ask for affirmation or opposition. The former polls produce similar results while the latter polls show the most extreme findings in favour and in opposition to the Government's plans.
So while none of the polls on gay marriage are invalid, the methods used by each of them need to be serously taken into account before conclusions are drawn in news reports.
UPDATE (13 June 2012)
Lesbian, gay and bisexual charity Stonewall yesterday released the findings of an extensive YouGov poll on public attitudes to LGBT people and on the Government's proposals over same-sex marriage.
The findings were reported accurately by, among others, the London Evening Standard:
"70 per cent of Britons are in favour of allowing gay marriage."
Again, however, it is worth bearing in mind the wording of the question posed by YouGov:
"The Government intends to extend the legal form and name of civil marriage to same-sex couples [Do you strongly support, support, oppose or strongly oppose]."
Unlike the other polls we have seen, the question adopts the term "civil marriage" rather than simply "same-sex marriage" or even "gay marriage". Moreover, it describes the policy change as an 'extension' rather than, for instance, a 'redefinition'. Evidence suggests that the choice of wording here can have a significant impact on the outcome of a poll.
Finally, it should be noted that the fieldwork for the poll was conducted in November and December last year, and attitudes may have changed since in light of the increased media coverage the issue has received in recent weeks.
For these reasons, we can't necessarily say that Stonewall's poll offers the best measure of public support for same-sex marriage today. It does serve to further show how important the question asked is to the results that are achieved.
Update 14 August 2012
On Saturday the Independent claimed that:
"a majority of people want the Government to ignore the results of its public consultation exercise and press ahead with its plan to legalise gay marriage."
This comes from a ComRes survey on behalf of the Independent which, according to the paper, found that:
"54 per cent want civil same-sex marriage to go ahead even if the idea is opposed during the consultation, while 37 per cent disagree and nine per cent are "don't knows"."
Clearly this is not the same as supporting gay marriage or otherwise and, reassuringly, we have not come across any media outlets trying to use the findings in this way.