The value of bad research
Pym Cornish explains why research with sample and measurement biases can still be useful - provided that the user is fully aware of them
NO DOUBT ADMAP READERS are far too sensible to bet on the National Lottery - with only half the stake money going into the prize fund, the theoretical odds are terrible.
But the results are not without interest. Every week there should be four or five jackpot winners, each winning a couple of million pounds. But in fact in one week there were well over a hundred jackpot winners, each winning a pittance, while in about one week in four the jackpot has not been won at all.
The explanantion is that people believe some numbers to much more random than others. In the lottery, where all combinations have an exactly equal winning chance, the answer is to bet on unpopular combinations - your chance of winning is unaffected but your prize may be many times larger. You can easily work out for youself which popular combinations to avoid from the variations in the weekly prizes, and restore the odds to your favour.
The Wayside Pulpit is not setting up as a tipster. To market research, the relevance of the lottery is that it shows that people do not understand probability and, hence, have difficulty in telling good and bad quantitative research apart. The reason for raising the subject is that there is now rather a lot of 'bad' market research around.
Three of the failings that make market research 'bad' are high or unrevealed sampling error, sample bias and measurement errors or biases. In the case of sampling error, the weakness is typically that the sample design and the weight scheme are not taken into account, so that the magnitude of the sampling error is under-estimated.
Sample bias arises where the achieved sample does not represent the universe being studied, and where relevant variations are not corrected by weighting. It can be particularly dangerous in quota surveys, since it will often be assumed that the quotas will control bias within cells as well as between them. In addition, bias of course results in systematic rather than random errors and, unlike sampling error, it cannot be estimated on internal survey evidence.
Lastly, measurement errors can arise in many ways. They include poor question wording, unsuitable questionnaire routing or layout and, perhaps most important, the use of inappropriate techniques. For example, audience measurement is very sensitive to technique: non-standard methods will always yield incompatible results.
The 'bad' research with which I am concerned is often a by-product of other marketing activities undertaken for non-research reasons. Examples include the collection of extra information from purchasers of durable goods, through guarantee cards, and from those who apply for financial or other services through their application forms. Another is the circulation of long self-completion questionnaires to the general public. In these cases the primary objective may be to generate mailing lists of people identified as having rare characteristics, but the resulting databases appear to be suitable for 'free' research analysis, without data collection costs.
An example of a different kind is inissue research for magazines or newspapers, usually with specialised subject matter. Here, research motives are predominant; the method is used because of the cost of conventional research, arising from the low incidence of readers in any form of representative sample.
When used for research, databases compiled in these ways are bound to be subject to severe sample bias. Response rates will almost always be low or very low. Response is also likely to be associated with some or many of the characteristics that are being measured, for example, with intensity of reading and involvement with the publication in the case of in-issue surveys.
Such questionnaires may often be complied without professional advice, resulting in ambiguous or badly worded questions. Topics that are inherently unsuited to self-completion techniques may also be covered.
These sample and measurement biases will not be corrected by conventional demographic weights, if weighting is attempted. It is therefore bias which is the danger with these kinds of 'bad' research rather than sampling error. Indeed, accumulating large samples is not normally a problem.
Research with these weaknesses is not without value, so long as the user is fully aware of them. In the first place, where studies with large samples are repeated identically after an interval of time, the time series trends observed are likely to be genuine. Secondly, in a publication field without objective research, comparisons between in-issue studies of different titles will be useful, so long as all of them are well-documented. Again, a user with special knowledge of his own brand or product category may be able to correct the flawed data adequately.
Unfortunately it is precisely those with least knowledge of the pitfalls, who are most likely to take 'bad' research at its face value and be misled by it. Like the average National Lottery punter they will find the odds are stacked against them.
The author is a director of RSL and a media and economic