IJMR Research Methods Forum

WARC reports from the
IJMR Research Methods Forum, London, 25 November 2008

From face-to face surveys to online ethics: MR challenges past, present and future
James Aitchison
1 December 2008

It was no coincidence that the Royal Society was the chosen venue for the International Journal of Market Research's first Research Methods Forum. Since 1660, this venerable institution has devoted itself to furthering progress and understanding in the physical sciences. It was high time, felt the organisers of the day, that the social sciences should strive for the same sort of certainties and universal acceptance.

Social sciences, however – and, in particular, the efficacy of many of the research methods on which they rely – are under increasing pressure. Technological change, social change, consumer change, cultural change and the complex (and itself changing) interplay between them have caused major questions to be asked of the social researcher’s traditional toolkit.

In the commercial field, those asking the questions are hard to ignore. It was, as the day's chairman Peter Mouncey reminded the audience in his opening remarks, a senior Procter & Gamble executive who predicted that “the consumer research industry as we know it will be on life support by 2012”. And it was an opposite number from Unilever, the other lead paymaster of consumer market research, who revealed that “I don’t know if we are going to have a choice but to move away from survey research.”

Add to that a couple of other factual twists of the knife from Mouncey, including MRS data from as long as 12 years ago reporting that almost a quarter of respondents did not enjoy participating in research, and the scene for the day was set.

But, as the proceedings unfolded, was the case for change as strong as the stage-setting opening session suggested? Well, new and innovative methods were certainly presented, such as Dunhumby’s interrogation of Tesco Clubcard data and a semiotic analysis of consumer websites. But it also became apparent that traditional methods – including the stalwart, survey research – were, if not alive and perfectly well, at least alive and very much still kicking. Add to that the myriad of new issues thrown up by the web 2.0 era, and the case is pretty much made for this Foum to become a much-needed annual event.

It’s a steal: face-to-face survey research

The latter observation was certainly supported by the keynote address from Professor Paul Wiles, the chief scientific advisor to the Home Office – at least in relation to the British Crime Survey, a random probability, face-to-face survey of 48,000 people annually.

But Wiles, whose self-styled job description was “to ensure that government research and data are good enough to base public decisions on”, conceded that there were problems and challenges facing government data-gathering activities and consumer research as a whole, such as:

Survey fatigue: the danger and tendency to over-survey the UK population
Methods: tools are not equipped for a mobile population, changes in media or ethnic diversity
Surveys: in the wrong hands, scientific standards are downgraded and “we become obsessed with statistics rather than evidence”
Digital: the cyber world leaves a data trail ripe for behavioural modelling

In conclusion, Wiles laid down the gauntlet in calling for the market research industry “to raise its game and move beyond churning out surveys”. It's a laudable aim, but maybe not immediately achievable in an era when commercial researchers are facing shaved budgets, tighter deadlines and increasing client demands.

Mixed-mode prospects

Sir Roger Jowell, a professor at CityUniversity, kept the focus on surveys with an overview of the European Social Survey, a biennial, face-to-face attitudinal survey in over 30 countries.

Face-to-face is applied in all countries to ensure equivalence, high response rates and the best quality data, and, in Jowell’s words, “its death is greatly exaggerated”. But there’s still mounting pressure in some participating countries for a switch. Cost is a key driver, with some stark differences: a research round that costs €3-4k in Bulgaria can cost €300-400k in Sweden.

Jowell’s colleague, Gillian Eva, presented some findings from pilot research into how a mixed mode approach might be applied and how problems of mode effects and continuity and comparability overcome.

Mixed mode experiments in Hungary and Portugal found:

Abstract and sensitive questions produce the greatest mode differences
Telephone data was most different from other modes
Telephone data produced more social desirability bias in its answers
Generally, a small proportion of items showed a small element of mode effects

The next phase of the research mapped different survey practices across Europe and, in particular, how telephone surveys affected response rates and quality. Shorter telephone interviews produced slightly higher response rates and data quality. The next phase of the research, currently being undertaken in the Netherlands, is designed as follows to see how more cost-effective modes could be introduced to the European Social Survey:

Mixed-mode issues were also the preoccupation for TNS’s Bill Blyth, whose “Changing respondent pathways” presentation mirrored his paper published in IJMR earlier this year, Mixed mode: the only ‘fitness’ regime? (accessible to WARC subscribers).

Picking up the themes raised in the previous presentation, he acknowledged that researchers have to consider more cost-effective, multi-mode survey designs, particularly for inter- and multi-national projects. But, argued Blyth in the paper and at the forum, coverage will be one of the largest sources of potential bias in any survey that uses any collection method other than face-to-face or mail.

Indeed, in a series of charts he showed the incomplete coverage of fixed-line telephones, mobile ownership and internet access around Europe, and the various forces that determined them (age, income etc), which culminated in this summary illustration of why no single mode can replace post or face-to-face:

However, Blyth said that mixed-mode approaches can provide better population coverage and, therefore, less sampling bias. But the flipside is that greater bias will be introduced from using different modes of questioning.

He argued that such issues will require a collaborative effort from the research industry to agree definitions, regularly measure coverage and share that data, develop mixed-mode sample designs and weighting best practice, and introduce survey design standards alongside existing process standards.

Turning the tables: transactional data and learnings from the Tesco Clubcard

In his opening remarks to the day, Peter Mouncey referred to a point made by Malcolm Gladwell in Blink, about how people are often ignorant about why they make the decisions they do, but never feel ignorant. In other words, the long-levelled criticism of survey research: people are unreliable self-witnesses.

The pair of presentations sandwiched between morning coffee and lunch deftly avoided any such accusations, since their focus was on the insights that could be mined from transactional data.

Mike Savage, a sociology professor from Manchester University, expanded on his Wither the Survey piece (accessible to WARC subscribers) that appeared in the IJMR earlier this year, in which he argued that the potential of transactional data should be better explored by social scientists.

He showed how visual analysis using new clustering techniques - illustrated using mobile phone network data from Finland - can map meanings from transactional and administrative data. A significant stumbling block at present is not just wider academia's somewhat sniffy attitude to commercial data - it's commercial operators' concerns over confidentiality.

Indeed, in the presentation that followed about data from Tesco's Clubcard loyalty scheme, Dunhumby’s Martin Hayward did quip that if he shared any data with government, "they'd probably leave it on a train!" (a reference to some of the government data security scandals in the UK during 2008).

In “Connecting the Dots”, he argued that analysis of the transactional data from the Clubcard programme enabled a far deeper understanding of Tesco’s customers than traditional survey research. Rather than ask people questions to predict their behaviour, he said, this approach analyses their behaviour to predict future needs.

He tracked the evolution of consumer research as follows:

We are what we earn – determined by social class
We are where we live – determined by geo-demographics
We are what we say – determined by lifestyle questionnaires
We are what we do – determined by granular data analysis

Such is the granularity of data that Dunhumby’s key premise is “there is no such thing as the average customer”. He gave the example of Tesco’s own-label Value and Finest ranges: everyday, 2.6 million shopping baskets include a Value item, 1.9 million carry a Finest item – and 1.2 million of these include both. So delineating – or even making the very case for the existence of – Value customers vs Finest customers is not straightforward.

Analysis of basket data enables Dunhumby to build a “DNA profile” on every customer, that enables customer insight, customer management and, ultimately, long-term customer loyalty. The next stage, he said, is to “join the dots” for a holistic understanding of Tesco customers.

To this end - and with slight irony after all the evangelism about granular, transactional data - he described Shopper Thoughts, a Harris-run attitudinal and behavioural questionnaire conducted among a panel of 65,000 Clubcard holders.

The old and the new

So it seems that survey research hasn't been abandoned quite yet, a fact reinforced by Richard Windle's post-lunch tour through the current methods used in media audience measurement. The boss of IPSOS Media CT arrived at a distinctly mixed conclusion as to whether technology is replacing interviews as the method of choice. In order of increasing technological influence, he summarised as follows:

Radio: maybe, but not yet
Print: maybe, but not clear how
Outdoor: it's starting to happen
TV: it's happened already, but not fully
Internet: yes, to a large extent

The agenda moved further afield with Gill Ereaut and Rachel Lawes' esoteric introduction to semiotic and linguistic techniques. They showed how art, popular culture and media is a hot-bed of "spontaneously occuring data" ripe for analysis and insight gathering.

One illustration they gave in an example-rich session was how the language and structure used in the shopping section of The Guardian newspaper's website helped determine, frame and define our attitudes to the activity:

The explosion of consumer-generated content on the web, they said, also presented researchers with a hugely fertile world of new data to explore. It was certainly an engaging session from two enthusiastic practitioners, but I still sensed a few doubters in the crowd. Rightly or wrongly, whilst these sort of techniques are perhaps no longer seen as the hippy wing of market research, they're still very much in the realm of complementary medicine.

The ethical network - or minefield?

Anjali Puri from Nielsen India continued the thematic drift web-wards with an overview of "webnography", and how best to track the levels and sentiment of consumer buzz in and around social media in its various guises. Along the way, she shared some interesting websites:

But things reached a real crescendo with the concluding session of the day - and, in particular, the discussion that followed.

Agnes Nairn's exploration of the ethical issues facing researchers in a virtual world threw up some particularly opaque hot potatoes, well-framed by this quote from a recent guest editorial in IJMR by Mike Cooke:

"If we can collect vast amounts of data passively without the individual being aware of the data trail they are leaving, does that give researchers the right to use it for other purposes?"

Mining consumer-generated content on the web certainly sits uncomfortably with existing MRS industry codes (e.g. B19: "Members must ensure that Respondents are able to check without difficulty the identity...of any individual...conducting a research project"), although it's fair to say that these were designed to govern a very different era.

However, that doesn't absolve the market research industry from its responsibility of defining what is good practice in the current era, but doing so will be far from easy. There was a clear split in opinion during the panel session. Web 2.0 research enthusiasts felt that information out there in the online public domain was fair game, whilst the more traditionalist wing voiced ethical concerns.

With pressure mounting on surveys, and the moral maze of next-generation methods becoming clear, market research is certainly living in interesting times.