Competitor Mystery Shopping: Methodological Considerations and Implications for the MRS Code of Conduct


Janet Dawson, and Jill Hillier
MSS/Research International
on behalf of the Market Research Society Professional Standards Committee

INTRODUCTION

Many organisations are currently buying competitor mystery shopping while others are considering it. Some will not presently contemplate it because of the potential problems it may pose, without adequate guidelines to ensure the quality of assessments of themselves as individual companies and their industry as a whole. Currently, there is lively debate concerning competitor mystery shopping and the direction in which the market research industry and its governing bodies should proceed. There are relatively few guidelines concerning the technique, and those that do exist (MRS and ESOMAR Codes of Conduct) leave some aspects open to interpretation and manipulation.

The MRS Professional Standards Committee (PSC) decided that it needed more information as to both agency and client views on competitor mystery shopping, to enable it to clarify its position and substantiate its decisions. The Committee, therefore, decided to conduct a survey on client and agency views of competitor mystery shopping.

The current situation

Mystery shopping as a research technique is growing in popularity. Just over two-thirds of companies who responded to this survey use mystery shopping, be it in their own company, in competitors companies or in both. Another 14% are planning to set up a mystery shopping programme in 1996. Therefore in future it is likely that over four-fifths of this sample will be using mystery shopping as a part of performance monitoring programmes.

Competitor mystery shopping is also relatively widespread; over one-third of the companies have competitor programmes in place at present.

The motor industry appears to be most advanced regarding mystery shopping. All of the automotive sector respondents were mystery shopping their own organisation and/or were engaged in competitor mystery shopping (see Table 1.1a).

TABLE 1.1a: COMPANIES CONDUCTING MYSTERY SHOPPING PROGRAMMES

Total Manufacturing Retail Financial Motor industry Public sector Other
Base - all (94) (12) (26) (28) (11) (1) (10)
% % % % % % %
Own company only 32 - 36 29 45 - 42
Competitors only 2 7 - 4 - - -
Own company and competitors 32 35 35 38 55 100 33
None but plan to 14 29 29 11 - - -
None, no intention to 14 29 - 18 - - 8
QNA 6 - - - - - 17

Looking again at the total sample, of those who conduct mystery shopping but not currently in competitor organisations, just under two-thirds are planning to set up such a programme within the next four years (see Table 1.lb). Therefore, it would seem likely that in four years' time, over half of all responding companies will be conducting competitor mystery shopping. This growth in competitor mystery shopping will probably occur mainly in the next two years.

TABLE 1.1b: THOSE PLANNING TO CONDUCT COMPETITOR MYSTERY SHOPPING

Base - all (28) - not 14% of 88 Total
%
In the next year 39
In the next 1-2 years 21
In the next 3-4 years 4
Never/unlikely 25

There are some companies which report that they do not intend to set up competitor mystery shopping programmes and these views should be taken into account.

The majority (68%) of companies conducting competitor shopping do not take part in a syndicate (see Table 1.lc). Very few syndicated studies are currently available. Only the financial industry has a syndicated study, in which approximately one-quarter of the companies who respond from that sector take part. The current extent and predicted growth of competitor mystery shopping, plus the lack of syndication, highlights the need for guidance and regulation.

TABLE 1.1c: NATURE OF COMPETITOR MYSTERY SHOPPING

Base - all conducting competitor mystery shopping (31) Total
%
Own competitor survey 68
Syndicated programme 29
QNA 3

However, even though 69% of respondents were aware of the current guidelines, it is disconcerting that 18% were unsure of what the codes were, that is, they did not know how to interpret them, and 11% did not know the codes at all. Knowledge of the MRS Code of Conduct varied by industry sector (see Table 1.ld). While the sample bases are small it would appear that the most knowledgeable groups are the retail and financial sectors, with those in manufacturing the least knowledgeable.

TABLE 1.1d: KNOWLEDGE OF MRS CODE OF CONDUCT MYSTERY SHOPPING CLAUSE

Total Manufacturing Retail Financial Motor industry Public Other
Base - all (94) (12) (26) (28) (11) (1) (10)
% % % % % % %
Known 69 57 82 75 64 100 75
Not sure 18 14 11 21 18 - 17
Not known 11 29 7 4 18 - 8
QNA 2 - - - - - -

The relative uncertainty of the clients indicates that if new guidelines are recommended and implemented within the Code of Conduct, they must be communicated as fully as possible to clients as well as agencies. This is particularly relevant when one considers that a large proportion of mystery shopping research buyers and users are from outside the market research industry. It is the responsibility of both parties to uphold the Code of Conduct and act in a way that does not detract from our credibility as an industry.

Main objectives

The main objectives of the study undertaken were to:

  • Identify what levels of competitor shopping are considered acceptable and unacceptable within the industry;

  • Identify the major differences between the different industry sectors;

and ultimately to:

  • Recommend acceptable and practical standards to the Market Research Society.

Method

The PSC/MRS identified 28 agencies conducting mystery shopping in the 1994 MRS Organisation Handbook. The number of agencies offering mystery shopping in the 1995 MRS Handbook is now 187. All were contacted by PSC/MRS and asked to canvass their clients for their views on competitor mystery shopping.

These agencies were sent a questionnaire and an example of a letter to send to their clients. They were asked to contact as many clients in as many different sectors as possible.

The questionnaire covered the following:

  • Type of organisation

  • Whether conducting mystery shopping and/or competitor mystery shopping

  • Whether competitor mystery shopping survey was syndicated or not

  • Whether the current MRS Code of Conduct was known

  • What they would consider an unreasonable amount of time might be for an assessment

  • How many competitor assessments of their organisation or a single outlet they deemed acceptable

  • Whether there were any other restrictions with regard to competitor mystery shopping in their industry

  • Any other comments.

Respondents were given every opportunity to explain their answers by a series of open-ended questions. Responses to these gave more depth to the findings and are used to illustrate the results in a qualitative way.

Sample

As we relied on the agencies themselves to distribute the questionnaires, a response rate cannot be calculated as it is not known how many questionnaires were sent out by agencies to their clients. Of the 88 replies we received one was from an agency. Hereafter this response will be included in the total sample but cannot be analysed separately.

Table 2.1a details the response broken down by industry sector. Most industries were well represented, the exception being the public sector, probably due to the small amount of mystery shopping activity in this area. Respondents were most commonly from the financial and retail sectors as they are currently the sectors most likely to use mystery shopping.

TABLE 2.1a: SAMPLE INDUSTRY SECTOR

Base - all (88) Total
%
Financial 32
Retail 30
Manufacturing 14
Motor industry 13
Public sector 1
Other 11

From the general comments made about mystery shopping (see Table 2.1b), it seems that mystery shopping as a technique is well known to respondents and is considered a useful tool:

Excellent test of what really happens. Objective.

Very valuable management tool.

Excellent technique to provide information on your own service performance as an improvement tool at a local level.

TABLE 2.1b: GENERAL COMMENTS ABOUT MYSTERY SHOPPING

Base - all (88) Total
%
Useful tool 22
Good, but should be done properly 11
Should be used with other methods 3
Other 22
QNA 42

However, a few respondents mentioned that care should be taken to ensure that it is conducted properly:

Needs to be regulated.

..needs careful planning, especially ensuring spread of when visits are made, and the way the findings are reported.

It should be encouraged with the appropriate safeguards.

The extent of qualitative comment here indicated that the respondent base was well placed to give their opinions on competitor mystery shopping.

Acceptable lengths and frequency of visits

Having looked at the current situation, this section investigates what length, and frequency of assessments, companies consider acceptable when mystery shopping competitors.

Length of assessment

The length of assessment viewed by respondents as unacceptable is shown in Table 3.1a. Just under four-fifths of respondents thought assessments should be no longer than half an hour, and of those just over two-fifths thought 10 minutes was too long. While keeping in mind that the sample bases are small, in all sectors the majority of respondents wanted competitive visits to be half an hour or less. In most sectors respondents were evenly divided between those who wanted visits restricted to less than 10 minutes and half an hour. However, in the manufacturing sector nearly three-quarters of respondents were in favour of visits being less than 10 minutes.

TABLE 3.1a: LENGTH OF COMPETITOR MYSTERY SHOP CONSIDERED 'UNACCEPTABLE'

Total Manufacturing Retail financial Motor industry Public Other
Base - all (88) (14) (28) (28) (11) (1) (12)
% % % % % % %
Under 10 minutes 43 72 36 36 46 - 42
10 minutes to half an hour 36 14 32 42 27 100 50
Half an hour plus 14 14 18 11 27 - 8
QNA 7 - 7 - - - -
Other - - 7 11 - - -

Respondents were asked to give reasons as to why they gave a particular length of time as the maximum acceptable (see Table 3.1b). Many respondents did not want the assessment to be any longer as their business and their real customers would be affected (acknowledging maybe that they were likely to be on the receiving end of mystery shopping as well as the instigator). Many also pointed out that if the assessment were any longer, the mystery shopper would not be acting as a normal customer.

TABLE 3.1b: REASONS FOR STATING UNREASONABLE AMOUNT OF TIME

Base - all (88) Total
%
Depends on nature of business 27
Any longer would affect business in a negative way 24
Do not like idea of competitor shopping at all 8
If a purchase made, it is okay 6
Other 15

Some individual comments were:

Time is money.

That is the maximum time it takes to carry out a shop. Any more time and they would not be normal customers.

Anything taking longer than 10 minutes is taking time away from staff who could be dealing with genuine customers generating business for the company.

Taking scarce resources away from genuine customers - possible implications for customer service.

Many respondents stated that the acceptable amount of time really depended on the nature of the companys business, a fair comment that might limit legislation on the issue. Some examples are as follows:

Approximate duration of a meal turnaround is 60 minutes, hence this time should be experienced (Leisure sector - Other).

To shop a dealership and carry out a thorough job could take up to half an hour (Car manufacturer).

Most transactions in fuel retailing take less than five minutes (Retail sector).

This depends very much on the instance; 20 minutes in a car showroom may not be too long, three minutes in a post office may be too long.

A couple of respondents in the motor industry and financial sector mentioned that follow-up contact after the assessment should be avoided in the competitor situation as this takes up even more time.

Some stated that they would be happy if a purchase was made (as long as it was not returned to the outlet). As might be expected, these companies were mostly in the retail sector. The main messages seem to be:

  • Respondents are worried about the effect on their business of lengthy competitor assessments
  • Acceptable lengths of visits vary by industry sector
  • The mystery shopper should not take up any more time than a normal customer
  • A purchase is not unwelcome when mystery shopping competitive retail organisations
  • Where a purchase cannot be made (e.g. bank, car showroom, telephone service), assessment lengths should be kept to a minimum and mystery shoppers should be making tentative enquiries rather than positioning themselves as a serious sales lead.

Frequency of assessment

Respondents were also asked how many competitor assessments, to both one outlet and the entire organisation, were acceptable in a three-month period. Many respondents reported uncertainty and hence the answers are less clear here than to some of the other questions.

The vast majority of those who felt able to answer wanted the number of assessments to any one outlet in a three-month period kept to a minimum (fewer than five to one outlet).

The views on the number of assessments acceptable to the whole organisation were mixed with 25% stating that the number should be five or fewer and others (10%) stating it should be over 30. We have been unable to analyse these results by sector due to the large number of dont knows in response to this question.

Again respondents were worried that if the frequency of visits was greater, their real customers would be affected and their business would suffer (Tables 4.1a and 4.1b):

This would jeopardise service levels for real customers.

Time is money and the opportunity to invest in a customer.

Impact on business.

TABLE 4.1a: FREQUENCY OF VISITS ACCEPTABLE TO ONE OUTLET VERSUS THE WHOLE ORGANISATION IN A 3-MONTH PERIOD

Total - Outlet Total - Organisation
Base - all (88) % %
None 6 7
One to five 56 25
Ten to fifteen 3 6
Twenty to twenty-five 2 8
Thirty - 1
Other (more than 30) 2 10
N/A/Don't know 31 43
Mean number 3.71 7.88

TABLE 4.1b: REASONS FOR STATING FREQUENCY

Base - all (88) Total
%
Disruptive to business if any more frequent 25
Any more would be unnecessary/do not have to visit all outlets 20
Depends on complexity 13
Must make a purchase 3
Okay if syndicated 2
Other 8
QNA 29

Some respondents pointed out that there was no need for all the organisations outlets to be assessed, but a reasonable number of evaluations should be conducted, so that the data would be reliable and robust:

Sufficient information needs to be obtained to provide comparisons, but should not be unduly intrusive.

As might be expected, others stated that the frequency of assessments depended on the complexity and length of each evaluation:

It would depend on how much time the visits took.

It depends crucially on the length of time involved and the nature of the shopping. If it were to involve 10 minute visits, quite happy for 10 visits per store over three months (i.e. one per store a week).

Again, retail companies stated that, ideally, purchases should be made:

Provided that time wasting doesnt exceed five minutes and that a purchase is made, competitor visits made once a quarter to a single outlet would be acceptable.

In summary, it is unnecessary for all the outlets in an organisation to be mystery shopped by a competitor, otherwise customers and the business may suffer. However a significant number should be conducted to provide robust data.

There could be some guidelines linking the frequency of assessments to the length of the visit, but again this may depend on the sector and number of total outlets in any one organisation included. As before, there would seem to be no all embracing answer but only answers that are dependent on several criteria.

Other restrictions

Respondents were also asked if they had any other comments on competitor mystery shopping and if they felt there were other restrictions in their industry.

Of those who gave further comments on mystery shopping, the majority agreed with competitor mystery shopping, although some specifically stated that it must be conducted properly:

It does provide valuable insight but needs careful planning, especially ensuring spread of when visits are made, and the way the findings are reported.

There is too much of it in an unco-ordinated, uncontrolled way. Too many cowboys, any commercial means to get the information are used without regard to impact on organisation shopped or the shopper (if a real person, or use of their data).

We must keep the quality standard high or this useful technique will fall into disrepute.

Other comments given are shown broadly grouped together in Table 5.1a.

TABLE 5.1a: OTHER COMMENTS ON COMPETITOR MYSTERY SHOPPING

Base - all (88) Total
%
Valuable, beneficial 18
Should be done properly 10
Difficult to interpret rules 2
Logistics sometimes difficult 2
Taping should be banned unless permission gained 2
Other 6
QNA 60

The restrictions on the industry sectors were not existing regulations or rules within the sector, but rather what the respondents felt was moral in terms of disruption to their business (Table 5.1b). A concern for sound business practice and not acting in any way which could be construed as subversive, would seem to underlie much of what was said here.

TABLE 5.1b: OTHER RESTRICTIONS IN INDUSTRY SECTOR

Base - all (88) Total
%
Wrong to take up too much time 10
Should not do anything out of ordinary 8
Should make a purchase 7
Wrong to place orders and cancel 5
Do not want mystery shopping 2
Other 10
QNA 58

Conclusions and recommendations

The findings have highlighted a need for clearer guidelines regarding competitor mystery shopping to avoid the disruption of an organisations business. The current guidelines are not as widely known or understood as they could be. Increased communication regarding their existence and their implications for the future is vital to the, still embryonic, mystery shopping technique.

Competitor mystery shopping should ideally be regulated to uphold the technique as a useful market research tool and avoid it being conducted in a free for all environment.

The findings show differences in views according to industry sector that could usefully be taken into account when devising guidelines. These differences demonstrate that there may not be one answer for all industry sectors (in relation, for example to frequency and the length of assessment) but rather, different answers depending on which industry sector is involved.

Some specific suggestions arising from the survey are:

  • Retail mystery shoppers should purchase an item where possible (particularly in the retail store environment) and not spend longer than a normal customer would in an outlet (as a guide, competitors could use their own companys average length of visit or telephone call).
  • Manufacturing: where a purchase is not possible, visits should be kept to a minimum - 10 minutes.
  • Motor industry: maximum 30 minutes per visit - maybe reconsider the necessity for a test drive; try to ensure that the salesperson does not have to take further time-consuming action, i.e. having to order a car for viewing from another dealership, follow up the mystery shoppers, etc.
  • Financial industry: if conducting transactions, up to half an hour only may be acceptable. Evaluations should involve a simple enquiry and should not normally involve a financial advisor/consultant.
  • Other industries: should be a normal customer enquiry and not take longer than a normal enquiry; ensure no follow-up. Maximum here 30 minutes.
  • Frequency of assessments: should ensure a spread over time; not all in one week or one day; should not be at busy periods, with visits to individual outlets restricted to a maximum of once a quarter.

We believe these findings go some way to indicating a need for mystery shopper guidelines within the MRS and the promotion of those guidelines. In addition the views of the buyers and users will help to construct a set of guidelines which will aid the industry in conducting high quality and ethical mystery shopping research.<