As the UK heads to the polls on Thursday, WARC's Alan Saywood explores why pollsters got the 2015 result so wrong, and asks what they must do to put things right and work toward the accuracy that the media and the public demand.
Pollsters in the UK are working to restore credibility in the eyes of the media and public after facing heavy criticism following the surprise results in the 2015 General Election and the 2016 EU Referendum. A new study, published in the International Journal of Market Research, shares the results of Ipsos MORI's investigation into what went wrong in 2015, and also considers the findings of an industry-wide enquiry sponsored by the Market Research Society and British Polling Council. With just a few days till election day, it will be interesting to see if these findings have improved polling accuracy for the General Election 2017.
The MRS/BPC enquiry found that the error was not in undercounted "shy Tories" or a late swing but rather in a response rate bias leading to the politically disengaged being under-represented. In particular, they critiqued pollsters for systematically under-representing Conservative voters.
However, this was not the main issue for Ipsos MORI. Roger Mortimore, director of political analysis at Ipsos MORI, and co-authors, Robert Worcester, founder of MORI; Paul Baines, Cranfield University; and Mark Gill, King's College London noted that in the Ipsos Mori final poll Conservative voters were actually overestimated (24.7% against 22.4%). The problem was that they overstated Labour voters ever more (24.1% against 18.5%).
They argue that while Sturgis' report makes some sound conclusions it fails to account for why the polls were largely correct in 2010 and 2005. As they put it: "to understand why knowing what went wrong is not the same as knowing how to put it right, it is first necessary to understand the nature of opinion polling intended to forecast election results." It is important to accept that polling is far more imprecise than what the media and public demand, so to reduce future errors the data gathered must be interpreted after the fact.
The authors argue that over-representation of the Labour vote cannot be assumed to be a sampling bias unique to 2015, even though methodologies hadn't changed between 2010 and 2015. Mortimore et al. found that the difference was "a change in the likelihood of voting responses" which showed a higher level of expected turnout, particularly among Labour voters. This led the authors to the conclusion that "Since it was the turnout answers of Labour supporters that changed, the likeliest explanation is that in 2015 Labour supporters were more prone to exaggerate their likelihood of voting than they had been in 2010."
So, how can the pollsters make it right? The authors argue that the primary need is "to tackle the under-representation of the politically disengaged in our polls." Ipsos MORI's initial response has been to attempt to control population characteristics that are related to engagement, primarily looking at newspaper readership and educational attainment. Factoring these in creates efficiency issues due to the harder to achieve sample size and therefore costing issues but may help counter the problems seen in 2015. Another solution has been to look at including other questions that measure aspects of political engagement to better gauge a respondent's likelihood to vote.
However, the authors warn that there is no a priori way of judging the success of these new models. In the 2016 EU referendum, applying these lessons gave a more representative sample but still provided a more incorrect projection than without them (due to misestimating the level of turnout by young working-class voters). Referendums are difficult for pollsters, as they lack enough comparative precedent and if anything serve to highlight the broad range of issues pollsters must confront going forward. As the authors themselves conclude, the analysis of polling methods and polling itself requires "judgement as well as science".
Read the full article: Forum: BPC/MRS enquiry into election: Ipsos MORI response and perspective
