The Warc Blog

The Warc Blog

Mythbuster: The spurious authority of numbers
 
Mythbuster, Les Binet and Sarah Carter, DDB
 
Mythbuster

Les Binet and Sarah Carter get a little bit angry about some of the nonsense they hear around them... like the spurious authority of numbers.

The other day we did something we hadn't done for a very long time. A young client, new to quantitative testing, asked us to go through his pre-testing questionnaire for a proposed new ad. Did it cover the issues he was interested in he wondered?

It was an illuminating and faintly horrifying exercise. Like most people in this business, we spend a lot of time in advertising research debrief meetings. Slick PowerPoint charts flash up, the numbers looking scientific, authoritative and objective. But peer behind the curtain, and things look much less impressive.

For a start, if this was real science, researchers would be trying hard to replicate real viewing conditions. But our questionnaire research scenario bore no resemblance to real life watching. The ad was an animatic not a real ad, which respondents were to watch on their computer, alone and paying close attention to. Some would be invited to film themselves on their webcams – the mechanics of this making the situation even more odd, with respondents instructed as to how to sit, arrange their hair and acceptable lighting conditions. Real people, meanwhile, watch ads in distracted, 'lean back' mode, wearing onesies, relaxing with their families.

And if this was real science, then there would be some attempt to replicate real buying decision-making in the questionnaire. This was an FMCG brand, where a shopper probably spends less than four seconds on the buying decision in-store. But with 60 questions to answer, this research interrogation would take far, far longer, with the respondent bored witless by the end.

And that matters a lot. Psychological research has repeatedly shown that people's choices and answers to questions are heavily dependent on how much time you give them. Ask for a quick answer and you'll get an intuitive, 'System 1' response. Ask them a series of considered questions and you'll get a more rational, 'System T response.

A brilliant demonstration of this is the 'Wilson Spooner Jam Experiment', where respondents were instructed to taste a range of fruit jams. Asked to decide quickly which jam they liked, ordinary people were just as good as food critics at picking good jam. But when asked detailed questions about sweetness, colour, and so on, their intuitions crumbled and they made poorer choices.

So, given that our 'test ad' was for a product chosen quickly in an intuitive 'System 1' way, the long and exhaustive 'System 2' questionnaire may well give misleading feedback. Not least because of the nature of the questions asked. Clearly many had been cut-and-pasted from other questionnaires. But our brand was unusual, and many of the questions didn't apply to it at all. Add the fact that the questionnaire writer was obviously not a native English speaker, and the result was a tedious and baffling mess guaranteed to confuse and annoy the most patient of respondents. And that matters too. When people are bored and annoyed by the questionnaire, they transfer those feelings to the brands and ads in question.

So, even our most cursory glance at the questionnaire raised serious doubts about this research. But who considers all of this when presented with the spurious precision and certainty of polished charts of numbers?

And it led to us musing; isn't it odd that clients and agency people often attend qualitative groups to listen to respondents 'first hand', but never do we remember anyone asking to experience quantitative research in action. And why are qualitative discussion guides more likely to be pored over than quant questionnaires? Maybe, to misquote Bismarck, numbers are like sausages – it's better not to see them being made.


This article originally appeared in the May 2013 issue of Admap. Click here for subscription information.



Subjects: Marketing, Consumers

07 May 2013 09:24
 

There are 1 comments on this blog

(Want to have your say? Add your Comment)

Les, Sarah - worrying stuff, but not unexpected. Plenty of quant . questionnaires are long, tedious.....System 2 is a flattering term....and the results are treated seriously whilst the respondent has perhaps checked out mid-way through the questionnaire..... My sense is there risk-aversion behind this, tick-box stuff. And maybe a lack of senior insights folk with the authority to front up changes to stuff that has been running for decades. The longer it goes on, the worse for all concerned, I fear. Maybe time to create a forum amonst MR clientside professionals - I am one - to address root-causes.
Edward A. 02 June 2013 at 9:18pm
Comments IconAdd your comment here:
Email :  
Forename :  
Surname :  
RadEditor - HTML WYSIWYG Editor. MS Word-like content editing experience thanks to a rich set of formatting tools, dropdowns, dialogs, system modules and built-in spell-check.
RadEditor's components - toolbar, content area, modes and modules
   
Toolbar's wrapper 
 
Content area wrapper
RadEditor's bottom area: Design, Html and Preview modes, Statistics module and resize handle.
It contains RadEditor's Modes/views (HTML, Design and Preview), Statistics and Resizer
Editor Mode buttonsStatistics moduleEditor resizer
  
RadEditor's Modules - special tools used to provide extra information such as Tag Inspector, Real Time HTML Viewer, Tag Properties and other.
   
 

Blog Search

Archives

  • 2014
    • August (18)
    • July (25)
    • June (22)
    • May (23)
    • April (20)
    • March (14)
    • February (10)
    • January (5)
  • 2013
  • 2012
  • 2011
  • 2010