How we use AI to help fact check party manifestos

As part of our efforts to fact check the 2019 general election, we’ve been taking a look at the manifestos of the main parties. In the run up to an election, many voters turn to impartial fact checkers like Full Fact to help them decide how to vote. Who can we trust? Which manifesto promises really are too good to be true? 

Manifestos typically contain many hundreds of claims from broad statements and ambitions through to detailed, fully-costed pledges. To help inform voters as they make up their minds who to vote for, Full Fact have analysed the manifestos of the larger parties and fact checked the important claims. While we can’t verify whether predictions will come true (even with AI!), we can often provide background information and relevant evidence in a clear and impartial fashion. But first, our fact checkers need to sort through the claims and decide which are feasible to check and which are the most important.

So how can we review these manifestos quickly and thoroughly? 

At Full Fact, we have been developing new technology to support our fact checking. Recently, we have developed an AI tool that can detect and classify different types of claims. It produces a list of every claim found in each manifesto with labels showing what type of claim it is e.g. a quantitative claim, a prediction about the future etc. Our team of fact checkers then uses this list to organize their analysis of each manifesto. For example, during their initial review they use filters to highlight only the claims that are most likely to be directly checkable. Later, they could use the filters to show only the predictions and promises being made about the future. While fact checkers usually can’t check these, we can put them into a clearer context to help voters better understand the implications of manifesto promises.

Example claims from this year’s manifestos

The Conservative manifesto promises to “Get Brexit Done”, a phrase which appears 22 times in the document, including this example:

We will help SMEs to become exporters, so that they can seize the opportunities that will become available once we get Brexit done.

Our AI tool tags this as both a “prediction” and a “correlation or causation” claim; the sentence is both a statement about the future (“We will help…”) and a claim about the consequences (“so that they can...”). Our fact check provided the context that “Getting Brexit done” is a process, not an event: Boris Johnson is not wrong to say that his deal will get Brexit done, but it is simplistic.

The Labour manifesto includes the claim that: 

There are 100,000 staff vacancies in NHS England, including a shortage of 43,000 nurses.

This sentence is tagged as a quantitative claim by our tool. Note that these figures are approximately correct but by themselves may be misleading as it doesn’t mean that no one is doing those jobs. NHS Improvement has previously said that between 90-95% of these vacancies were being filled by temporary staff. 

The Liberal Democrat manifesto states:

Staying in the European Union will secure a £50 billion Remain Bonus, with the economy two per cent larger by 2024-25.

This is classified by our AI tool as being both a ‘prediction claim' and a 'quantitative claim'. As we’ve written in our fact check, this is a fair assessment of the best available forecasts. But those forecasts themselves contain a high degree of uncertainty, so it’s too definitive to say remaining in the EU “will secure” a £50 billion bonus. 

Finally, consider this sentence from the SNP manifesto:

More than a decade of austerity, years of low wage growth and a freeze on social security payments, has left families struggling and our public services stretched.

The AI tool classifies this as having claims that are both “quantitative” and “correlation or causation”. The phrase “more than a decade” refers to a quantity, and the sentence claims that families struggles are caused by the austerity, low wage growth and social security freeze. Our full fact check of the SNP manifesto can be read here.

Using AI to classify claims

In common with many machine learning tasks, we needed a labelled set of examples to train our AI tool to do what we needed. We started by defining a claim as “any statement about the world that is either true or false.” We then defined nine different types of claims after internal discussions and experimental prototyping. These include quantitative claims that refer to a specific value in the present or past; predictive claims that are about the future; and personal claims, that recount personal experience. Earlier this year, staff at Full Fact manually labelled nearly 5000 sentences according to what types of claims each sentence contained. 

After collecting this sample, we used it to fine-tune a pre-trained model called BERT. BERT is a tool released by Google Research that has been presented with hundreds of millions of sentences in over 100 languages. This makes it a broad statistical model of language as it is actually used. It can be adapted to a wide variety of specific tasks. In our case, we fined-tuned it to recognise the different types of claims made in news articles, TV news subtitles -- and in manifestos. We estimate our claim-type classifier has an accuracy of around 75-80%, broadly in line with results from similar NLP tasks by other researchers.

The table below shows the relative sizes of the seven manifestos that we analysed in depth (in alphabetic order). “Claim count” is number of sentences containing at least one claim. 

Conclusion

With a typical manifesto using over 20,000 words to make around 900 claims, fact checkers need all the help they can get with navigating and selecting claims to check. Presenting a complete list of claims to a busy fact checker in a clear and uniform way can help by itself. But adding the claim types can also help the fact checkers to focus on claims that are most worth checking, for example by allowing them to pick out quantitative claims or temporarily hide predictions. In each case, the full text of the manifesto is still available to check the wider context of the claim. Our fact checkers can now use the same tool to identify important and checkable claims from a broad range of media sources as part of Full Fact’s day-to-day fact checking process too.

 By providing a timely scrutiny of manifestos, we hope to encourage all political parties to write more accurate and carefully-considered manifestos in the future. More immediately, we hope our analysis of the key claims helps us all make more informed choices as we head to the polling booths.