Mythbuster: The spurious authority of numbers

Les Binet and Sarah Carter

Les Binet and Sarah Carter get a little bit angry about some of the nonsense they hear around them... like the spurious authority of numbers.

The other day we did something we hadn't done for a very long time. A young client, new to quantitative testing, asked us to go through his pre-testing questionnaire for a proposed new ad. Did it cover the issues he was interested in he wondered?

It was an illuminating and faintly horrifying exercise. Like most people in this business, we spend a lot of time in advertising research debrief meetings. Slick PowerPoint charts flash up, the numbers looking scientific, authoritative and objective. But peer behind the curtain, and things look much less impressive.

For a start, if this was real science, researchers would be trying hard to replicate real viewing conditions. But our questionnaire research scenario bore no resemblance to real life watching. The ad was an animatic not a real ad, which respondents were to watch on their computer, alone and paying close attention to. Some would be invited to film themselves on their webcams – the mechanics of this making the situation even more odd, with respondents instructed as to how to sit, arrange their hair and acceptable lighting conditions. Real people, meanwhile, watch ads in distracted, 'lean back' mode, wearing onesies, relaxing with their families.