Advertising Research Foundation Workshop
Advertising Research Foundation
641 Lexington Ave., New York, NY 10022-4503;
Tel: +1 212 751 5656 Fax: +1 212 319 5265
|
Advertising Research Foundation WorkshopAdvertising Research Foundation 641 Lexington Ave., New York, NY 10022-4503; Tel: +1 212 751 5656 Fax: +1 212 319 5265 |
October 1998 |
David Napior
Roper Starch Worldwide
and
Jane Bailey
Time Inc.
What is the net buying effect produced by advertising? This is perhaps the most vexing question facing advertising, media and research professionals in the late 1990's. But it is not a new question. In fact, it was a question that Daniel Starch and his associates, Howard Stone, Stan Sargent, Morgan Neu, and Mel Weiss, tackled over a period of twenty years from 1944 to 1964. More than 400,000 personal interviews were conducted and readership and product purchasing evaluated for 20,000 ads in the development and testing of NetAPPs (an acronym for Ad Produced Purchased.
NetAPPs is the crown jewel of Daniel Starch's valuable contributions to advertising research. Unearthed from the Starch archives in the early '90s, NetAPPs has since received renewed interest in the last two years thanks, in large part, to the efforts of Professor John Philip Jones. Jones, at Timothy Joyce's suggestion, reviewed the Starch NetAPPs manuscripts and discovered striking similarities between his own work on measuring short-term advertising strength (STAS) and Starch's NetAPPs.
The conceptual foundation for NetAPPs was Starch's belief that measurement of advertising effectiveness was best accomplished by statistically comparing the buying behavior and attitudes of persons essentially alike in all significant respects except for the fact that some had been exposed to advertising while others had not He came to this conclusion by rejecting two alternative hypotheses.
First, Starch believed that analysis of the relationship between total sales of a product and advertising weight is inadequate to the task of measuring advertising effectiveness. There are simply too many complex factors affecting total sales. For a product with no established customer base, advertising-induced purchases will almost certainly result in rising sales volume. For a product with many established customers, brand switching or demographic trends leading to a reduced need for the benefits provided by the product class may result in fully effective advertising merely sufficing to maintain existing levels. This is a common situation for well-established, big-share-of-market products. For some products facing heavy competitive advertising and promotional activity, effective current advertising may simply slow down the rate of sales decline.
In addition to rejecting changes in total sales as a barometer of advertising impact, Starch also believed that direct questioning of consumers regarding the impact of advertising on their purchase behavior is unlikely to yield a valid measure of advertising impact. His argument was that very few consumers are sufficiently attentive to the multiplicity of forces driving them to purchase products to be able to report them accurately on a retrospective/ introspective basis.
Initially, Starch measured NetAPPs by isolating two groups of people:
Buyers of a brand who have read an issue of a magazine containing an ad for it and who have perceived (that is, noted, associated, or read most of) that ad.
Buyers of a brand who have read the same magazine issue and not perceived that ad.
John Philip Jones calls this NetAPPs approach the Single-Issue Readership Method (SIR) method. Starch calculated SIR-NetAPPs by:
This arithmetic produced what Starch viewed as the relative impact of magazine advertising on consumer buying behavior.
In the final six years of the NetAPPs research program, Starch adopted a somewhat more conservative approach to measuring NetAPPs:
John Philip Jones calls this the opportunity-to-see (OTS) method, since ad readership does not play a role in the computation of NetAPPs, merely the opportunity to see the ad. The calculation of OTS-NetAPPs is quite straightforward – it is simply the ratio of the OTS buying rate to the non-OTS buying minus one, expressed as a percentage. The OTS method is closest in most respects to John Philips Jones STAS methodology and has a desirable inferential robustness. However, it is less reliable, in the statistical sense of the word, than SIR-NetAPPs and it does not incorporate directly into its calculation, as SIR-NetAPPs does, the ad readership phenomenon, which Starch believed was the essential precondition to advertising impact.
Although Starch used a variety of research designs in the course of the development and validation of NetAPPs, the basic approach was essentially quasi-experimental survey research using in-person data collection methods. The literature of statistical inference is replete with warnings regarding possible threats to the validity of causal inference from quasi-experimental survey designs. NetAPPs was not immune to such issues and Dr. Starch addressed the bulk of the more salient problems in a thorough and prescient manner in his unpublished 1961 manuscript 'Measuring Product Sales Made by Advertising.' Included in that manuscript are discussions of knotty validity-related problems, such as:
Starch was particularly concerned with threats posed to the validity of the NetAPPs methodology by possible ambiguities in establishing the direction of flow of cause and effect from single-source survey designs. He acknowledged that the flow of cause and effect between ad reading and product buying action might be from reading to buying or from buying to reading. Some readers buy the product because they had read the advertisement and some buyers read the advertisement because they buy the product.
From a philosophy-of-science point of view, Starch was of the opinion that it did not make a great deal of difference which way the sequence runs – from reading to buying or buying to reading. In his words:
Daniel Starch
Nonetheless, Starch did conduct a number of NetAPPs studies in which he collected a respondent's purchasing data both before and after ad exposure occurred. Using an analytic method involving matching of the control and test group samples with respect to pre-test purchasing behavior, he was able to establish that the bulk of the NetAPPs causal flow went from readership to buying and not buying to readership.
From 1944 to 1964, Daniel Starch and his associates – Howard Stone, Stan Sargent, Morgan Neu and Mel Weiss – conducted an extensive program of experimental and quasi-experimental research for development, refinement and testing of the NetAPPs procedure for evaluation of print advertising impact. Ad readership and product purchase studies were conducted for every issue of The Saturday Evening Post from July, 1944 to July, 1964 and for every issue of Life from 1946 to 1955. In all, more than 400,000 personal interviews were conducted and readership and product purchasing evaluated for 20 000 ads.
During the six-year period from 1959 through 1964, Dr. Starch developed his final refinement of the NetAPPs method. He collected extensive purchase data for 73 brands from more than 110,000 respondents when issues contained advertisements for the brands as well as when they did not Over 700 different advertisements were evaluated in this last six years of NetAPPs. It was from this dataset that Starch was able to apply his OTS approach to NetAPPs measurement.
Dr. Starch's central, summary finding was that 'about 14% of net additional purchases were stimulated during the week following the appearance of an advertisement which attracts 30% perceivers.' Of course, there was considerable variation about this central NetAPPs tendency. Much of the variance, however, was systematically related to measurable co-variates, such as the stopping-power or persuasiveness of the ad – the greater the persuasiveness the higher the NetAPPs, in general.
One way Starch supported the scientific validity of the NetAPPs method was by pointing to the overall consistency of findings in a wide variety of application settings, such as estimation of:
In addition, Starch supported the NetAPPs method by noting the consistency of his findings other with independent empirical observations culled from the many studies he and his associates conducted over the years for the Starch Ad Readership and Starch Magazine Readership services:
The Starch team also addressed the selling power of magazine advertising from a return-on-investment (ROI) perspective. This was accomplished by supplementing the estimates of net ad-stimulated purchasing with information about the total cost of the advertising concerned and the dollar-value of the NetAPPs. For example, in one series of NetAPPs studies, Starch examined toiletry brand Y, which had 16 one-page advertisements in magazine M. These advertisements had an average Starch readership score of 29% and NetAPPS per average-issue-reader (AIR) per advertisement of .0209, with an average of 39 cents revenue per purchase. Thus net ad-produced revenue per AIR per advertisement was estimated to be $.0082. On the other hand, the cost per ad insertion per AIR was estimated from independent sources to be $0.0032 per page per average-issue-reader. The return on investment in advertising was computed as net ad-produced revenue divided by the cost of advertising, or 257%.
Table 1 summarizes these computations, step-by-step. Dr. Starch reports that he applied the above procedure for determining return on advertising investment in 177 cases for which the required data was available. He found, on average, that net-ad-produced purchases per dollar of advertising cost amounted to $3.04. This is admittedly a naïve approach to ROI analysis, but it does seem to have a certain heuristic clarity.
Advertising ROI Calculations |
|
| Pages of Advertising per year in magazine M | 16 |
| Starch ad readership score per advertisement per average-issue-reader (AIR) | 29% |
| Net ad-produced purchases per advertisement per AIR | 0.0209 |
| Price-per-purchase | 0.39$ |
| Net Ad-produced revenue per AIR (0.0209 x $0.39) | 0.0082$ |
| Ad cost per AIR (from Starch readership reports) | 0.0032$ |
| ROI (=83/32) | 257% |
In early 1998, Time Inc. commissioned Roper Starch to evaluate the effectiveness of advertising to be placed in Sports Illustrated for a major packaged snack producer. Roper Starch Suggested that the problem could be addressed by applying the NetAPPs single-source methodology. We appended to five standard Starch Ad Readership studies of Sports Illustrated a series of purchase and purchase-intent questions for eight major brands in one of the packaged snack product sub-classes. Only two of the eight brands were advertised in the five issues of SI tested, and never both in the same issue. The NetAPPs tack-on questions were:
In total, we had an in-tab sample of 462 adult males for this NetAPPs pilot test. A cross-tabulation of the purchase behavior and intention data by ad readership and OTS is presented in Table 2.
Bought SNACKS in last month |
Bought test brand last week |
Intend to buy test brand in next month |
||||||
| Brand A Had OTS |
271 | 100% | 165 | 61% | 8 | 3.0% | 44 | 16.2% |
| Noted Did not note |
181 90 |
67% 33% |
116 49 |
64% 54% |
6 2 |
3.3% 2.2% |
33 11 |
18.2% 12.2% |
| No OTS | 191 | 100% | 132 | 69% | 8 | 4.2% | 32 | 16.8% |
| Brand B Had OTS |
191 | 100% | 132 | 69% | 14 | 7.3% | 50 | 26.2% |
| Noted Did not note |
90 101 |
47% 53% |
66 66 |
73% 65% |
8 6 |
8.9% 5.9% |
27 23 |
30.0% 22.8% |
| No OTS | 271 | 100% | 165 | 61% | 12 | 4.4% | 52 | 19.2% |
| Combo Had OTS |
462 | 100% | 297 | 64% | 22 | 4.8% | 94 | 20.3% |
| Noted Did not note |
271 191 |
59% 41% |
182 115 |
67% 60% |
14 8 |
5.2% 4.2% |
60 34 |
22.1% 17.8% |
| No OTS | 462 | 100% | 297 | 64% | 20 | 4.3% | 84 | 18.2% |
We have calculated NetAPPs by both the OTS and SIR methods for both test brand purchase and purchase intent. The results of these computations are seen in Table 3.
Last-week Buyers |
Next-month Purchase-Intenders |
|
| Brand A OTS method |
(29.5%) | (3.1%) |
| SIR method | 24.7% | 24.7% |
| Brand B OTS method |
65.5% | 36.4% |
| SIR method | 19.0% | 13.0% |
| A/B Combo OTS method |
10.0% | 11.9% |
| SIR method | 12.0% | 12.5% |
There are some interesting consistencies between our abbreviated 1998 implementation of the NetAPPs method and Starch's pioneering magnum opus:
John Philip Jones, in his Telmar Awards paper, provides a very good summary evaluation of the Starch NetAPPs methodology.
The SIR method has the practical advantage of being ... an extension of Starch's ongoing Ad Reading and Noting service. This means that a ready made evaluative service can without too much trouble be offered to the advertising industry. There is also a very real benefit in being able to factor readership into the calculations. The effectiveness of print advertising has two separate dimensions which the SIR system enables us to untangle:
The NetAPPs system combines the methodological virtues of simplicity and comprehensiveness with its core contribution, the provision of useful quantitative assessments of magazine advertising effectiveness on a timely and cost-effective basis. Although the increasingly persistent call for demonstrable 'accountability' often seems unfair to those of us in the industry who know in our heart that magazines represent, at the very least, a viable advertising medium, it must, nonetheless, be answered. NetAPPs was developed in a time when the power of print was taken for granted and, as a result, was probably ahead of its time. In this new age of accountability, NetAPPs may have the opportunity to finally realize its full potential.
 http://www.warc.com | © Advertising Research Foundation |