Comparing Response Rates and Response Content in Mail versus Electronic Mail Surveys

Raj Mehta
and
Eugene Sivadas
University of Cincinnati


Introduction

Traditionally, researchers have relied extensively on mailed questionnaires to collect information from respondents (Dillman 1978). Consequently, a large body of knowledge has been generated in the various social science disciplines on innovative ways to improve overall response rates and data quality in mailed questionnaires (see Church 1993; Kanuk & Berenson 1975; Dillman 1978; Linsky 1975; Yu & Cooper 1983 for a review of literature). Research has also focused on the strengths and weaknesses of different methods of gathering information (e.g. mail versus face-to-face) e.g. Alan & Alan 1994; Krysan et al. 1994). However, very few studies have attempted to evaluate newer information technologies as a way of collecting data. Blattberg & Glazer (1993) describe information technology as an information infrastructure that dramatically increases the amount of information that can be processed and transmitted in almost real time. Thus, this technology allows sequences of one-way communication between people to be replaced by meaningful two-way communication. One of the many important aspects of this technology is that it allows users of the technology to communicate with each other through electronic mail (e-mail). E-mail, simply, is a way for computer users to exchange messages (text, pictures, computer programs, audio, and video) between distant computer users as long as there are networks such as the internet and internet service providers such as CompuServe connecting them.

As e-mail becomes more accessible, the possibility of collecting data from large sample surveys from respondents on a computer network becomes a distinct possibility. The median income and education of US e-mail users are well above the US average (Lefton 1993). Thus, e-mail may become a method of choice to conduct surveys of upper to middle-class subjects. In fact, a few researchers (Kiesler & Sproull 1986; Sproull & Kiesler 1985; Walsh et al. 1992) have suggested that e-mail may become a routine research tool (like regular mail).

In their 1986 study Kiesler & Sproull received similar responses to close-ended questions from e-mail and mail questionnaires. Unfortunately, their study was limited to students and staff at a single organisation (Carnegie Melon University). Clearly, if e-mail is to become a routine research tool, there is a need for further studies that (1) establish the feasibility of broader data collection through e-mail and (2) calibrate e-mail surveys with regular mail surveys. The objective of this study is to compare mail and e-mail surveys on various dimensions. We conduct an experiment, where respondents on a large global network (the internet) are sent mail and e-mail questionnaires.

E-mail and regular mail

E-mail is essentially an electronic version of regular mail. Both have a number of similarities. Like regular mail, every individual has an unique e-mail address. Also, just like regular mail, to send a message one composes a letter and sends it to that individuals e-mail address. The computers on the network take care of the transmission and delivery.

Researchers have to make a number of trade-offs when they decide on a data collection method (Dillon, Madden & Firtle 1993). Important factors that influence the choice of a method are: speed; cost; sample control; convenience; response rate; and quality of data (Dillon, Madden & Firtle 1993).

Speed

An e-mail message can be sent across the world in a matter of seconds. Similarly, replies can flow back just as rapidly. Regular first-class mail can take much longer, on an average from two to three days within the US to seven to ten days internationally. Though the researcher may find it quicker to send e-mail, it may not necessarily be quicker to receive responses, since not all e-mail users check their e-mail daily.

Cost

Once respondents have access to a network, the marginal costs of collecting and communicating data electronically are much lower than costs of interviewing, telephoning, and sending questionnaires through the mail. These savings may be even more substantial if the respondents are scattered worldwide. For example, the minimum postage for a first-class letter costs 29 cents (now 32 cents in the US) and 50 cents (for international) whereas e-mail (domestic or international) is free for most users of the internet. However, with growing talk of commercialisation of the internet, these cost advantages may or may not accrue over the long term.

Sample control

Typically, in any mail survey, a researcher has little control on who actually completes the questionnaire once it has arrived at the respondents address. E-mail provides more control than regular mail in that most users read their own e-mail. Usually colleagues or secretaries at work do not read and answer someone elses e-mail (Dyson 1993). Individuals with e-mail access tend to guard their electronic mailbox (Dyson 1993). Thus, a survey intended for an individual is more likely to be read and answered by that individual.

Coverage error

Unlike regular mail which can be used to reach most people in the population, access to e-mail is much restricted. E-mail may be limiting in that it is appropriate for sampling more educated upper to middle income respondents since they form the bulk of current e-mail users. The sampling frame for electronic surveys is restricted to members of the population who have access to computer networks and to those who feel comfortable using them. However, in recent months, this frame has been expanding rapidly. Take for example, the internet, the largest electronic network. Many sources (Dern 1994; Hahn & Stout 1994) estimate the number of users on the internet as close to 15 million. This number is growing rapidly at approximately 10% a month or doubling every few months. The total size of this WorldNet or mail internet is estimated as 30 million. At this rate over 100 million people worldwide will be connected to the WorldNet by the year 2000. Clearly, e-mail users may not be representative of the population since socio-economic status, age, experience and access to computers may restrict access to e-mail.

Convenience

Once a file (questionnaire) is created it can be e-mailed easily. All that is required is to type the respondents e-mail address. Anyone who has struggled through a mailing can appreciate the time saved (e.g. time spent on photocopying, stuffing envelopes, and addressing). E-mail also involves personnel costs in that e-mails have to be e-mailed individually to the respondents.

However, a copy of all outgoing mail can be saved in the mailbox. This makes it extremely convenient for repeated communications with the respondent, such as follow-up reminders or clarifications (i.e. there is no need to retype e-mail addresses). Another convenient feature is that all questionnaires that come back have the time and date stamped on them. Also, like everything on an e-mail system, if one writes an incorrect address, the system alerts almost immediately.

Response rates

E-mail questionnaires may be more difficult to complete. It is much easier to circle a number in a paper-pencil exercise than to use the cursor keys to position a cursor and then enter a response on an electronic survey. Monetary incentives increase response rates (Church 1993). It is not possible to provide monetary incentives via e-mail to respondents. The interactive nature of e-mail makes it easier for potential respondents to refuse to participate in a survey or even complain to the offending researcher. Additionally, many e-mail users have to pay for the volume of e-mails they receive. Respondents may thus react less kindly to unsolicited e-mail surveys.

Response content

Another difference between mail and e-mail is the social aspect of e-mail. Kiesler & Sproull (1986) in their study found that responses of the electronic respondents reflected more candour. This supports earlier research (Kiesler et al. 1984; Sproull 1985) that suggests that electronic respondents tend to be both more self-absorbent and uninhibited. This induces them to write longer answers to open-ended questions. However, e-mail provides less anonymity to the respondent in that the receiver can in most cases identify the respondent from the header.

Method

Overview

Respondents were randomly assigned to one of five groups to assess the feasibility of gathering information using e-mail survey. To calibrate our study, we send mail questionnaires to the first group (group 1). These respondents received a mail questionnaire without pre or post-notification. No monetary incentives were provided. This is the simplest data collection method.

A number of past studies (Church 1993; Dillman 1978; Kanuk & Berenson 1975; Linsky 1975; Yu & Cooper 1983) have shown that (1) pre-notification, (2) monetary incentives, and (3) post-notifications and follow-ups result in higher response rates and better response quality in mail surveys. Thus, the next group (group 2) of respondents receive pre and post-notifications. Also, monetary incentive of $1 was enclosed with each survey.

In the e-mail versions of the survey (group 3 and group 4), we replicate the conditions in groups 1 and 2, respectively. Thus respondents in group 3 receive an unsolicited e-mail questionnaire and the respondents in group 4 receive an e-mail questionnaire after they have shown a willingness to participate in the study through an initial polite e-mail requesting their participation. All the respondents in groups 1,2,3, and 4 were residing in the United States. Finally, to group 5, we sent e-mail questionnaires to non-US respondents after soliciting their participation through prenotification.

All respondents received a five-page (300 lines) 19 multi-itemed questionnaire. In all instances, the sponsoring organisation was a large Midwest university. The questionnaires were constructed such that the e-mail questionnaires would look the same as the mail questionnaires. Three different appeals were included in the cover letter: a social utility appeal that emphasised the worthiness of the research; an egoistic appeal that stressed the respondents place and importance; and finally, an appeal to help the researchers (Yu & Cooper 1983). As an incentive, the respondents were asked to write their address (mail or e-mail) if they wanted to see the results of the study.

Respondents

Along with e-mail, Usenet is one of the most popular services within the internet. It is a global community of electronic bulletin board systems (BBSs) or simply, it is a large collection of about 5000 discussion groups each centred on a specific topic (e.g. humour, investment). People from all over the world participate in these discussion groups by posting an article of interest to a specific newsgroup.

We wrote a program that collected the e-mail addresses and signatures of all the people who posted articles on 20 most popular newsgroups. We ran this program for a period of one month (from 02/01/94 to 03/01/94). At the end of the month we had collected e-mail addresses of over 6000 different users. We then used systematic random sampling (every tenth address) to draw a sample of 663. Based on the e-mail address of the poster, we separated the domestic (US) and foreign posters.1 In our sample we had 491 US users and 172 non-US users.

Next, using the e-mail addresses, posting organisations, and the signatures as a basis, we determined the postal addresses of the domestic respondents. As indicated earlier, we then divided the US respondents in four groups. Two of these groups (groups 1 and 2) received mail questionnaires and the other two groups (groups 3 and 4) received e-mail surveys.

Table 1 provides a summary of treatments for various groups.

TABLE 1: TREATMENT FOR VARIOUS GROUPS

Group 1

Group 2

Group 3

Group 4

Group 5

Method Mail Mail E-mail E-mail E-mail
Geographic scope US US US US International (countries outside the US)
Prenotification No Yes No (unsolicited e-mail questionnaire) Yes (initial e-mail requesting participation in the study Yes (same as group 4)
Customisation (Dear Internet user or Mr/Mrs name) No Yes Yes Yes Yes
Method of choice for receiving a questionnaire Mail (no option) Mail (no option) E-mail (no option) Choice of e-mail, fax, and mail Choice of e-mail, fax, and mail
Monetary incentive None $1 None  None None 
Post-notification No Yes No Yes Yes

 

The survey instrument

The respondents to the study are internet users who read one or more of the diverse newsgroups on the internet. In recent years there has been a lot of debate on the internet regarding its commercialisation. We thus constructed a questionnaire that questioned the respondents on their attitudes towards this commercialisation.2 The first section asked for the respondents ability and frequency of use of various internet services (classification variables). The second section asked about their attitude towards the use of these services by commercial firms (attitudinal variables). It had 23 closed-ended questions and one open-ended question. As suggested by Hansen (1980), the open-ended question afforded us an in-depth analysis of the quality of the responses as assessed through their verbosity.

Dependent measures

The effectiveness of a survey method for collecting data can be measured using response rate, speed of response, response completeness, response bias (comparison) and response quality as dependent variables (Hansen 1980).

Results

The responses from all five groups are tested for (1) cost, (2) response slowness, (3) response rates, (4) degree of completeness and (5) response bias. Table 2 gives the complete history of the experiment. Table 3 provides some key results.

TABLE 2: STUDY HISTORY

Group 1

Group 2

Group 3

Group 4

Group 5

Sample size 202 107 60* 122 172
Address unknown 10 4 none 3 12
Refusal NA NA NA 14 15
Said yes to participate NA NA NA 79 129
Method Mail Mail E-mail 72 e-mail, 2 fax, 5 mail 127 e-mail,
 2 mail
No response to any e-mail messages NA NA NA 26 16
Incomplete or empty questionnaires 1 2 2 None None
Usable responses 88 85 24 75 103
Number of responses by the end of day 1 None None 8 28 44
Number of responses by the end of day 2 None None 17** 31 55**
Number of responses by the end of day 3 None None 18 37** 56
Number of responses by the end of week 1 8 None 24 60 87
Number of responses by the end of week 2 39 41 24 67 90
Number of responses by the end of week 3 65** 64** 73 100
Number of responses by the end of week 4 75 65 24 73 101

* Discontinuous at N = 60 due to complaints
** Median number of questionnaires received this time frame

TABLE 3: KEY FINDINGS: RESPONSE RATE, RESPONSE SPEED AND COST

Group 1

Group 2

Group 3

Group 4

Group 5

Response rate 45% 83% 40% 63% 64%
Response speed parameter b(1) 0.96 1 0.6 0.24 0.32
Response speed parameter b(2) 0.89 0.78 NA 0.45 0.5
Response speed parameter b(3) 0.87 0.72 NA 0.42 0.61
Minimum cost per questionnaire 58 cents $2.16 Free Free Free

Note: smaller values of b imply quicker response times

Cost of study

From Table 3, it is clear that the e-mail versions of the study were extremely inexpensive. The per respondent cost for sending questionnaires to group 1 and 2 were $0.58 (2*0.29) and $2.16 (0.29 + 0.29*2 + $1 + 0.29), respectively. This does not include additional administrative costs (e.g. copying, stuffing envelopes). Sending questionnaires to a few respondents (groups 4 and 5) who requested surveys by mail or fax did cost the researcher. Only two international respondent requested a mail survey. Just the cost of mailing out the survey to these respondents was $0.95 each. Thus at present, e-mail which involved no additional costs appears to be a cheaper method.

Response speed

Response speed during the first few days provide evidence for quick responses through e-mail. As Table 3 indicates, more than half of all completed e-mail questionnaires were received in two or three days. In contrast, it took three weeks to receive half the completed questionnaires from the regular mail groups. Thus, on an average mail questionnaires take about 10 times longer to return (2 days versus 21 days).

We use the following model adapted from Huxley (1980) as a way to measure response speed:

Rt/N=1 bt          (1)
where:

Rt = number of responses accumulated by the end of week t,
N = number of questionnaires mailed initially
t = the elasped time in weeks from initial mailing to the end of week t, and
b = parameter to be estimated empirically.

The parameter, b, is of special significance. By definition it takes the values between 0 and + 1. A high value of b (closer to 1) shows a slow response speed; low value of b (closer to zero) shows a rapid response rate. Huxley (1980) suggests using equation (1) to estimate b with a single data point at the end of each week. A lower value of b during initial weeks (e.g. weeks 1, 2, and 3) would indicate quicker response rates and higher values would indicate slower response rates. The lower values of the parameter b at the end of weeks 1, 2, 3 indicate that the e-mail responses were much quicker than the mail responses. Although, this may seem intuitively obvious at the onset, the results indicate that respondents tend to: (1) read their e-mails regularly and (2) respond more quickly to an e-mail. A two to three-day median response for e-mail compared with a three-week median response for mail cannot be explained away by a statement such as e-mail is naturally faster. It is obvious that mail questionnaires remain on the respondents desks for a longer period of time.

Response rates

Typically, the conservative method for computing response rates for mail surveys is: response rate = total number of usable responses/(total sample size minus undelivered) (Dillman 1978). The response rates were 45%, 80%, 40%, 63%, and 65%, respectively (see Table 3). As these numbers indicate, group 2 (regular mail with pre and post-notification and monetary incentive) had extremely high response rates. International and domestic e-mail surveys with prenotification were next in terms of response rates.

Response completeness

As in Hansen (1980) the mean number of unanswered questions was calculated as a measure of response completeness for the closed-ended questions. Over 92% of all returned questionnaires were completely filled out. The average number of unanswered questions by groups was very close to zero in all instances (0.0145, 0.0004, 0.041, and 0.0113) 3. The international group had similar averages (questionnaires completely filled out = 93% and average number of unanswered questions = 0.006).

In the case of the open-ended question, the response to that question was treated as a yes-no variable (i.e. the respondent either answered or did not answer that question). The open-ended questions require the respondents to list the names of the newsgroups they read most frequently. Surprisingly, we saw hardly any difference between the average completed responses for both the open- and closed-ended questions. The average completed responses for the open- and closed-ended questions were 95% and 98%, respectively.

We compared the actual number of listed newsgroups across the four domestic groups. The average number of newsgroups listed were 4.12, 4.57, 4.60, and 4.42, respectively. There was no significant difference across groups (F = 0.95, p <0.4). The international group also had a similar mean (mean newsgroups listed = 4.75, t = 1.12, p = 0.26). Thus, there are no differences across groups with regard to the number of items (newsgroups) listed in the open-ended questions.

Response comparison

Again, we use the method described by Hansen (1980) to compare responses across the groups. We compare the distribution of responses of one group of respondents with that of the other groups of respondents.

Response patterns for 17 classification questions were compared across the four domestic treatment groups. Only one of the resulting chi-squares was significant at the 0.05 level. Because this number is smaller than would be expected by chance at this level, we conclude that there was no difference between the respondents in these four groups with respect to these classification variables. Further, we conducted a series of ANOVAs to compare 38 attitudinal questions across the treatment groups. After using conventional methods to correct for family-wise alpha (to reduce the possibility of getting significant results by chance), there was only one ANOVA that was significant. This provides compelling evidence that there were no differences in classification or attitudinal variables across the four groups.

We handled the international group slightly differently. Because the internet first became popular in the US, we expected significant differences between the domestic and international users with respect to the classification variables. The response patterns of these classification patterns were compared between groups 4 and 5. Five of the resulting chi-square statistics were significant at the 0.05 level. The results showed that the international users were newer on the internet and therefore used fewer internet services than the domestic respondents. Further, we conducted a series of t-tests to compare the 38 attitudinal variables between groups 4 and 5. Only one of the 38 resulting t statistics was significant. This provides strong evidence that although the international respondents are newer on the internet, they had similar views with respect to the commercialisation of the internet.

Response quality

We compared the average number of meaningful words in comments that the respondents wrote. The average number of words were 2.86, 3.84, 10.17, and 9.14. Clearly, the e-mail respondents wrote more comments than the mail respondents (F = 2.52, p = 0.058). In most of the comments, the e-mail respondents clarified their attitudes towards the commercialisation of the internet and explained their close-ended responses. Thus overall, the e-mail responses tended to be more insightful than the mail responses. As expected, the average number of comments for the international group was between the regular mail and the domestic e-mail groups (average number of comments, international = 7.825).

Discussion and directions for future research

The main objective of the study was to assess the e-mail survey method by (a) studying the feasibility of using e-mail as a tool to gather survey data and (b) comparing it with regular mail methods.

Clearly, e-mail surveys were much quicker and much less expensive. The majority of e-mail responses are received in two or three days compared with three weeks for mail responses. The minimum mailing cost for a mail survey was $0.58. There were no direct costs associated with e-mail surveys. These differences become even more critical when dealing with international respondents. A comparable international sample gave us similar responses without any additional costs in terms of time and money. E-mail allows an easy access to global respondents. In our study, we contacted over 150 respondents in different parts of the world. We received many completed questionnaires back from Europe, South Africa and Australia within a few minutes of sending them out. More importantly, it was possible quickly to answer or clarify questions that some respondents had. For example, the phrase significant other was used once in our survey. Two European respondents did not understand what we meant by the term significant other and indicated that in an e-mail. E-mail provided us with an opportunity to clarify such questions quickly and inexpensively, while the respondents were still working on the questionnaire. On the closed-ended questions the e-mail surveys provided similar results when compared with the regular mail surveys. The e-mail respondents wrote more clarifying and illuminating comments than the mail respondents. Thus, in some ways e-mail responses were more insightful. These in itself are some compelling reasons to use e-mail as a tool for data collection.

The main disadvantage with e-mail is that not everybody uses e-mail, which limits its usage in that may be more appropriate to sample middle to upper-class respondents.

Another disadvantage with e-mail is that respondents who use e-mail are extremely sensitive about their e-mail accounts (Dyson 1994). For example, many respondents who received the large 300-line unsolicited survey in their e-mail account were quite upset and contacted us via e-mail and conveyed their annoyance at receiving such a survey (group 3). Unlike regular mail, where respondents have no direct and inexpensive method of communicating their feelings towards receiving unsolicited questionnaires, here they also have the ability to communicate their frustration back to us. Because we received such complaints from respondents we decided to abandon that part of the experiment half way (after 60 questionnaires were e-mailed). Thus, unsolicited e-mail surveys are clearly unacceptable. Also, since many respondents pay by the amount of time they are logged on or the amount of internet e-mail they receive, this is an unfair burden on many potential respondents. Thus, it is both a good idea and sound practice to seek permission of an e-mail respondent before emailing a questionnaire. Respondents have to be given a chance to opt out. The disadvantage to this is that it allows respondents to self-select into the study. The response rates will be lower than through mail surveys that provide incentives but one is likely to get a more motivated respondent.

At present, there is no mechanism within e-mail to provide monetary incentives to respondents. If response rate is the researchers main concern and the researcher has adequate time and money, the best method is to allow time for pre and post-notifications and include a token monetary incentive. One may be able to pre-mail incentives by using regular mail. This may also circumvent the problems associated with unsolicited e-mail surveys. Future research should explore the impact of this method on e-mail response rates.

E-mail holds promise as a survey research tool. However, many important questions about the use of e-mail as a method of collecting survey data remain unanswered. This is just one study that assesses (1) the feasibility of e-mail questionnaires and (2) compares results with similar mail methods for a specific population. Clearly, other studies that include a broader sampling frame, multiple topics, various types of sponsoring organisations, and incentive levels would help develop a framework that describes the conditions under which e-mail will be most feasible.

ENDNOTES

  1. The last two characters of a persons e-mail address indicates the country where the poster has an account. Thus if the last two characters are us the poster has an account in the United States. As, initially, the internet was intended to be used mainly within the United States the last two characters (e.g. us) are not used for many of the US sites but are always used for the non-US sites. (Back)

  2. A paper is available with the authors that describes the results of the survey. (Back)

  3. Although these differences are significant (F 22.31, p<0.001), there is little practical significance to these differences since all means are so close to zero. (Back)

Acknowledgments

We would like to thank Norm Bruvold, Murali Chandrashekaran, Bob Dwyer, and James Kellaris for their comments on an earlier draft.

REFERENCES

Au, Alan & Tse, Alan (1994). A comparison of the effectiveness of mail and facsimile as survey media on response rate, speed, and quality. Journal of the Market Research Society, 364, pp 349355.
Blattberg, Robert C. & Glazer, Rashi (1993). Marketing in the information revolution. In Robert C. Blattberg, Rashi Glazer, & John D.C. Little (Eds.), The marketing information revolution. Boston: Harvard University Press.
Church, Alan H. (1993). Estimating the effect of incentives on mail survey response rates: a mets-analysis. Public Opinion Quarterly, 57, pp 66-79.
Dern, Daniel P. (1994). The Internet guide for new users. New York: McGraw-Hill Inc.
Dillion, William R., Madden, Thomas J. & Firtle, Neil H. (1993). Essentials of marketing research. Homewood, IL: Richard D. Irwin, Inc.
Dillman, Dow A. (1978). Mail and telephone surveys: the Total Design Method. New York: Wiley.
Dyson, Esther (1993). If nothing on e-mail is private, just say so. Computerworld, 27, February 22 pp33.
Hahn, Harley & Stout, Rick (1994). The Internet complete reference. Berkeley: Osborne McGraw-Hill.
Hansen, Robert A. (1980). A self-perception interpretation of the effect of monetary and nonmonetary incentives on mail survey respondent behavior. Journal of Marketing Research, 17, 1, pp 63 8.
Kanuk, L & Berenson, C. (1975). Mail surveys and response rates: a literature review. Journal of Marketing Research, 12 pp 440 53.
Kiesler, Sara, Siegel, Jane & McGuire, Timothy (1984). Social psychological aspects of computer-.mediated communication. American Psychologist, 39, pp 112334.
Kiesler, Sara, & Sproull Lee S. (1986).Response effects in the electronic survey. Public Opinion Quarterly 50, pp 40213.
Krysan, Maria, Schuman, Howard, Scott, Lisli Jo & Beatty, Paul (1994). Response rates and response content in mail versus face-to-face surveys. Public Opinion Quarterly, 58, pp 381399.
Lefton, Terry (1993). CompuServe to log on with Visa. Brandweek, 34, March 15, pp 4.
Linsky, A.S. (1975). Stimulating responses to mailed questionnaires: a review. Public Opinion Quarterly, 39, pp 82 101.
Sproull, Lee S. (1986). Using electronic mail for data collection in organizational research. Academy of Management Journal, 29, pp 159 69.
Sproull, Lee S. & Kiesler, Sara (1986). Reducing social context cues: electronic mail in organizational communication. Management ScienceD, 32, pp 14921512.
Walsh, John P., Kiesler, Sara, Sproull, Lee & Hesse, Bradford (1992). Self-selected and randomly selected respondents in a computer network survey. Public Opinion Quarterly, 56, pp 241244.
Yu, Julie & Cooper, Harris (1983). A quantitative review of research design effects on response rates to questionnaires. Journal of Marketing Research, 20, 1, pp 3644.<