Recently, we undertook a project with a lovely market research agency to understand the evolving aspects of the industry and the agency's place in it. Readers of Paid Attention may remember that it contains a chapter entitled "Why all market research is wrong" but also that I didn't say market research wasn't useful. Rather, I said that people rarely know why they do what they do and don't predict their own behaviour well, so claimed responses shouldn't be taken at face value.

We began the project by doing some research into the industry discourse from reports and news and through interviews. We spoke to twenty research practitioners, a dozen buyers of market research at large client organisations, a few agency CSOs, a few academics and a market research industry consultant. Some of what we learned was only tangentially germane to the project, but exposed some fascinating narratives inside what is now called the 'insight industry'. In the literature, we found this:

"The days of focus groups ... it's over. It's about observing behaviour. Big data comes together with granular understanding of human behaviour" - from Ann Mukherjee, president, global insights at PepsiCo.

This seemed interesting, since focus groups are the visible core of qualitative research. Thinking Pepsi was perhaps an outlier, we looked to contextualise the quote. Googling 'focus groups are dead' produced innumerable other claims like it.

Armed with this insight, we started speaking to people, some of whom do qualitative research, including with groups, and some of whom buy that research. We asked them if demand was declining for focus groups. Surprisingly, we heard that there was not a decline, but an increased demand.

We then dug into the PwC report commissioned by the Market Research Society on the industry. It's called The Business of Evidence, and the analysis of the market shows the two growth areas are: 1) Big Data analytics, which somewhat corroborates the quotation; and 2) qualitative research, primarily groups. (Meanwhile, ethnographic studies are expensive and time-consuming and we didn't speak to a single client that didn't have a reduced budget and even less time.)

So we had a conundrum. The discourse was at odds with the evidence gleaned from the report. The paradox was only resolved when we discovered a piece by Mindspace's Clare Wade, bemoaning the industry's detrimental conflation of 'qual' with 'focus group': "This is reflected by Diageo's launch of its Consumer Planning Team manifesto, 'Say No to Focus Groups', when what it really meant was, 'say no to doing qualitative research badly for the wrong reasons'."

The manifesto said 'just say no' because groups were being done like the drunkard using a lamp post for support, rather than illumination - a line Ogilvy appropriated without attribution from Andrew Lang. This sentiment from senior management was echoed across client organisations.

Around the turn of the century, the qualitative term 'group discussion' was nudged out by the American 'focus group', which often took things at face value, as opposed to the psychological approach embraced in the UK. The term was debased by politicians and ridiculed by the media. Books such as Consumer.ology by Philip Graves made focus groups seem methodologically suspect, and they were often undertaken by people without any training. But, no better, affordable way to glean certain kinds of insight has been found because, frankly, talking to people is a good idea. So insight managers were forced to rebrand groups as co-creation sessions, roundtables, workshops and the like to avoid accusations of being old-fashioned.

CSOs we spoke to bemoaned the fact that - because market research agencies now have their own clients with their own budgets - planners at advertising agencies no longer do qualitative research. This is a shame since, if one is to be the voice of consumers, it's a good idea to actually speak to some.

The 'death of x' business narrative is salesmanship, not strategy, and should trigger alarm bells whenever it's heard.

That said, we uncovered some interesting innovations that suggest an integrated future. The Greenbook Insight Innovation award winner this year was Remesh, a company that 'builds artificial intelligence to engage, empower and understand large groups of people'. Essentially, you define audience and it pulls a representative social media corpus and turns it into a bot that can have a conversation with a trained qualitative researcher.

In the future, then, perhaps all meaningful Big Data analytics will be qualitative.