Holier than thou about media research with holes in it

Neil Mortensen, Thinkbox's Research & Planning Director, questions the validity and conclusions drawn from some media research...

The time has come to draw a line in the sand about shoddy media research giving distorted and meaningless 'advice' to advertisers. Enough is enough.

Some background: recently Thinkbox published Payback 3 by the respected analytics company Ebiquity. This was a huge piece of research: an econometric analysis of 3,000 ad campaigns across nine advertising sectors between 2006 and 2011. It compared the impact of different media on sales and profit on a like for like basis, effectively measuring the behaviour of every household in the UK through what they bought and the media they consumed.

Payback 3 was commissioned by Thinkbox but conducted independently by Ebiquity. We announced we had commissioned the study in advance of receiving the results and as such held ourselves a little hostage to fortune; whatever results came back, we were committed to publishing them transparently. If TV was found to be the worst profit generator and, say, outdoor proved to be the most effective, then we would have been neatly skewered by our decision to invest in proving that advertising works (with our fingers unashamedly crossed that TV came out well).

Some relief followed the delivery of Ebiquity's analysis. It found – like the IPA's Marketing In the Era of Accountability and all other serious effectiveness studies that had gone before– that TV's ROI is not just better than others but is actually increasing – up by 22% over the last five years. They also found that TV creates the most profit and is 2.5 times more effective at creating sales uplift per equivalent exposure than the next best performing medium (press). Exact figures varied in different categories, but TV was particularly more effective for FMCG brands.

We issued a press release announcing the findings, presenting them as Ebiquity had given them to us , making sure that we emphasised the results for TV rather than indulge in any 'TV beats x' nonsense. The headlines of the press release were 'New study shows TV ads deliver the most profit' and 'TV ROI is 22% higher than 5 years ago, despite recession'. We deliberately didn't go with 'TV has a better ROI than radio, press, outdoor and online display'. Although this is what the study found it is simply not Thinkbox's style; moreover we made sure that we celebrated that all advertising broadly has a positive ROI.

Now, I'm well aware how holier than thou I sound and that maybe I'm setting myself up for a fall at some point, but I honestly don't care. What I do care about, however, is giving advertisers proper, well-grounded, thoroughly researched, trustworthy, actionable advice that they can use to improve their businesses. Ebiquity's study does this – even if you are tempted to take it with a cynical pinch of salt because Thinkbox paid for it.

So you can imagine my shock when I read this headline barely a week later about another new research study: Online beats TV for ROI.

Eh? Had Ebiquity got it wrong? In fact, not just wrong but diametrically wrong? Naturally we examined the study that had apparently proved this surprising claim. It is difficult to know where to start with research that appears to be so unfit for purpose and misleading but here is our understanding of it.

The research came from Gfk, sponsored by Google, and it had looked at – wait for it – all of 8 (eight) FMCG ad campaigns. It also based its findings on mixing two methods of data collection: half from a rigorous online meter and half modelled from the claimed behaviour of what appears to be an online survey.

This mixture is like comparing evidence from CCTV to an unreliable eyewitness statement taken sometime after the event. Using an online sample is, at worst, like asking people waiting in a station if they ever travel by train. It makes the research structured to find in favour of online.

It also based its ROI calculation solely on sales in the 2-4 week period after the campaign had finished. Do marketers only invest for a 4 week return? TV's advertising effects decay much more slowly than for other media which is partly the reason it delivers such great ROI.

There was also no apparent attempt to look at TV's multiplier effect on other media – particularly search - or the uplift it gives to a brand's wider product portfolio, all significant factors in Ebiquity's analysis. In fact, exactly how sales were attributed to each medium remains rather unclear but given that search was found to create the biggest sales uplift I fear we are back in the bad old world of 'last click wins'.

All partisanship aside, which piece of research would you trust? The one based on 8 campaigns or the one based on 3,000? The econometric analysis or the online sample?

Even if this was an isolated case, it would still be intolerable; but it isn't an isolated case – it is simply the latest example.

The Gfk/Google research isn't just shocking because it contradicts a much bigger and more robust study; nor is it shocking because the methodology appears to be so unfit for purpose. My chief criticism is that it is recklessly misleading at a time when advertisers need the best advice possible on where to invest their money.

So please can we all – clients, agencies, research companies, trade organisations, news outlets – try to kill off this worrying trend in frankly pointless research. Dodgy surveys are fine fodder for the tabloids on a Monday morning but they have no place in serious business discussions. Their sweeping claims stand on the thinnest of ice and serve to undermine the good research that is out there – and there is lots of that if you look.

We want you all to be deeply sceptical about every piece of research published and properly interrogate its approach rather than just swallowing it as a press release- yes, including for Thinkbox's own. We think our studies will withstand the sternest of your glares, but if they don't, fair enough. The danger is that the real insights will become obscured in a self-serving fog of misdirection.

And you're in danger of more blogs like this.