While the commissioning of qualitative and quantitative research is clearly important, the role of the focus group in creative development and evaluation is questionable, argue Vince Usher, Lead Brand Thinker and Adam Ferrier, Chief Thinker/Founder of Thinkerbell.

“[They were] judge, jury, and executioner.” It’s a popular idiom, first committed to paper in 1717 by Daniel Defoe as part of a memoir written about the Church of Scotland. With this phrase Defoe analogised the persecution and ultimately often execution of Church-affiliated citizens by regular everyday soldiers given, extraordinarily, magisterial powers. The purpose, according to Defoe, was “to swiftly quell what may have been the church’s potential rebellion, at the risk of a slow judicial process.”

It’s a wild-sounding proposition, but the concept really isn’t dissimilar from the ways in which our industry today uses focus groups to judge, analyse, and sentence to death creative ideas. The jury, laypeople, are ordained by those with the ultimate power, marketers, with the power to both judge creative ideas and swing the executioner’s scythe their way. Stepping back, do we really think this is good, brand-building, decision making? Or is it a smokescreen, fear thinly veiled, stamping out rebellion and maintaining the status quo?

The focus group in a literal sense is almost always eight or so bored people who’ve never met each other, sitting in a deliberately dull meeting room behind a two-way mirror. The concept also holds up metaphorically: unqualified people attempting to home in on an idea, on the focal point, to decide its worth, generally with naught more than dinner at the front of their minds. It’s wormed its way into all forms of culture, from politics to architecture, to movies, entertainment, academia and, of course, marketing – the logic being that ideas are best judged by those who’ll experience them. Dig a little deeper and this ends up being about as sensible as Defoe’s soldiers, there’s almost no independent research that supports this method. There’s a bit out there, of course, but most of it fails the sniff test once you look into the body or company that has funded and designed the research.

From the more rigorous investigations, we know that group decision making is often flawed at all levels. Cass Sustein’s (1999) research on legal juries showed that in jury situations groups often push moderate participants toward increasingly extreme positions until regression toward a confused and diffident mean takes place. Closer to home in the world of advertising Brooke Hempbill (2018) writes, paraphrasing the work of Professor Rachel Kennedy (Ehrenberg Bass), that “the real question is whether pre-testing can predict if a campaign will lead to the behaviours in-market the advertiser desires. The short answer is that in a lot of cases, it cannot.”

Creative agency scepticism has been omnipresent around the focus group, especially when used for ‘creative testing’. For years a video showing the famous Apple 1984 ad being exposed to a focus group has done the rounds, highlighting the absurdity of testing creative work in this fashion. 

And so, taking inspiration from Ehrenberg Bass and the makers of that video (Arnold Worldwide), we decided to test the focus group as part of a podcast series called ‘Black T-Shirts’.  

We asked two focus groups what they thought of one of the most lauded and successful ads of our generation: the famous John Lewis Christmas ad ‘Monty The Penguin’. As reported by the IPA in WARC (2020) the ads were very successful, returning 883% ROMI, £1.2bn in incremental sales, and £411m in profit.

A beloved classic in the UK, ‘Monty the Penguin’ was unrecognisable to an Australian audience. The ad was shown in two formats: one cell received a storyboard of the ad, the other cell received the ad as shown to television audiences in the UK.

The focus group that received the storyboard version of the ad (Cell A) viewed it as absurd and depressing:

  • “It’s a bit [censored]… I don’t see how it’s relevant, it doesn’t tug at my heart strings at all. I like penguins, I’d like to have a penguin, and I don’t understand what the point of it all is”
  • I “think the premise of the story was really lovely, but I think the animation, and the lack of colour in the animation made it not as engaging and the voiceover that read very slowly like you were reading to a child made you zone out and not focus on what was happening in the storyline”
  • “I feel sad, and I’d tell people not to watch it, because they’d get p*ssed off too”

Overwhelmingly, the group strongly opposed the making or progression of creative development of the ad.

Those who saw the final ad (Cell B) were not nearly as scathing, but still didn’t see the ad as something that would impact their behaviour:

  • “It doesn’t tell you too much about the brand, it’s more focused on the little boy and the penguin, I’d like to see more about the brand”
  • “It was that long of an ad and it took that long for me to realise that it was a department store ad, that yeah, I probably would’ve lost interest a long time ago…”
  • “Realistically I’m not going to base my buying decisions off an ad about a penguin, at least not consciously”

These results are interesting for several reasons:

  1. Many market research companies laud the fact that their benchmarks show no difference between a finished ad and a storyboard concept, but our research suggests  that this is much more likely salesmanship than a well-backed argument.
  2. Development of good, effective, advertising could be at risk if testing ads at a ‘storyboard’ concept continues (it’s hard to storyboard charm!).
  3. Without context consumers will reject, or at least view sceptically, the very best of advertising.

While we’re strong advocates of research and the commissioning of qualitative and quantitative research, there’s a lot that makes us cynical of the contribution of the focus group in creative development and evaluation. It’s also worth considering the serious dearth of independent research on pre-testing. More than anything, this article should be a stimulus for someone to take this on.

Sources

1. Arnold Worldwide (2008) 1984 Focus Group video for awards show

2. Wood, C. (2018). John Lewis: An amazing decade

3. Hemphill, (2018). Ehrenberg-Bass on the science of pre-testing campaigns.WARC.
Sunstein, C. (1999). The Law of Group Polarization, University of Chicago Law School, John M. 

4. Olin Law & Economics Working Paper No. 91

5. Defoe, D. (1717). Memoirs of the Church of Scotland, in Four Periods: I. The Church in Her Infant-state. II. The Church in Its Growing State. III. The Church in Its Persecuted State. IV. The Church in Its Present State

6. Gordon W. Admap February (1997) “See: Is the right research being used? Out of the Goldfish Bowl: can advertising research ever replicate reality?”