The trade press overflows with interesting predictions about the future of advertising.
But there's a problem.
Experts, marketing or otherwise, have an awful record of predictions. Philip Tetlock, a psychologist at the University of Pennsylvania, ran a 20-year study that analysed 82,361 forecasts from 284 experts. He found that their predictions were as likely to be wrong, as right. In his memorable phrase, the average pundit fared no better than 'a dart-throwing monkey'.
It's not just that these marketing predictions misguide us. There's also an opportunity cost. Our fixation with the future crowds out an interest in the past. Yet there is value in looking backwards to people who grappled with similar problem to ours.
So let's take a look at one current problem, turning data into insights, and look back to what we can learn from the 1940s. In particular, the experience of one man: Abraham Wald.
Wald was the son of a kosher baker and grandson of a rabbi, born in the Austro-Hungarian Empire in 1902. He was a mathematical prodigy – at primary school he was correcting his teachers and at secondary school he was correcting his textbooks.
After a glittering early career at the University of Vienna, his success stalled in the 1930s with the rise of anti-Semitism. In 1938 he fled to America. That decision probably saved his life. By the end of the war his whole family, bar one brother, had been killed.
Once in the US he worked for the government's Applied Mathematics Panel, putting his talents to the war effort. Specifically, how to reduce the death rate among European bomber crews. Theirs was such a dangerous job that about half of bomber crews died during the war.
Wald's team needed to determine which parts of the planes should be armoured. Too much and the bombers would be slow and cumbersome, easy targets for German anti-aircraft defences. Too little and the crew were exposed - a handful of hits might bring them down.
The researchers set to work collecting data. As planes returned from a sortie his team recorded which parts had been punctured by bullets. When hundreds of bombers had been logged, a pattern emerged. As the diagram clearly shows, the areas most regularly hit were the wings, fuselage and tail.
The top brass was ecstatic; Wald's methodology had convincingly identified the critical parts to armour. Case closed.
But was it?
Wald's interpretation was radically different. His bosses had fallen victim to "survivorship bias": assuming returning planes were representative of all aircraft. That was a mistake. The data collected omitted the crucial planes: downed ones. They held the clue to solving the problem, not the returning planes.
The parts peppered with bullet holes could be ignored on returning planes, after all they had limped home. It was the unscathed spots that needed armouring. Any plane hit in these areas was at the bottom of the channel.
Wald's counter-intuitive thinking was quickly tested and shown to improve survival rates. His approach became standard US policy until the Vietnam War.
This is more than a historical anecdote. It's a guide to how best to use data, whether in the military or marketing. There are four key principles that can be drawn from Wald's experience
Don't rely on intuition
Wald was a genius (a genuine one, not in the way we now bandy about the term) yet even he didn't rely on introspection alone. He collected data, and only then, developed his hypothesis.
Far too many plans and approaches are based on gut feeling alone. This is an issue because agency staff are not representative. A study by Newsworks investigated this problem by recruiting 30 young media planners to complete the IPA Touchpoints diary for a week. This meant that their media habits could be compared to the broader population.
The differences were stark. Take TV. Young planners watched half the volume of TV compared to the population as a whole. Other media showed a similar variation. This discrepancy is a concern because of what psychologists term the false consensus effect. It's the idea that we over-estimate how much others share our beliefs and behaviours.
Again this bias is prevalent in agencies. We tested this by asking Zenith planners to estimate what proportion of the population had an iPhone. We then crossed that data with whether they had an iPhone or not. Those who owned one thought half the population did, while those who didn't put the figure at just a third.
Relying on intuition is dangerous – it leads to plans that would influence us but not the consumer. Instead, every plan needs data at its heart. And Wald gives us guidance on how to achieve that.
Collect data in the simplest manner that answers the question
Wald's methodology required pen, paper and a numerate assistant. Basic even for the 1940s.
This is important for marketing as too often we obsess over the complexity of our methodologies: econometrics, machine learning, artificial intelligence and so on. It's as if we believe we can replace the hard work of thinking with the high costs of measurement.
But insights don't need high budgets. For a recent brief into a male incontinence drug we wanted to help the planners understand the target audience. We had no budget so we used a technique we call 'method planning'.
Over a weekend we texted the planners at random times. Each time they received a text they had to stop what they were doing and get to a toilet within two minutes. This helped the planners understand the experience of the target audience. From this experiment we uncovered two useful insights. First, incontinence is not a great concern when people are at home. After all, a toilet is a few seconds away. It's when they're out of home that it's a worry. This led us to recommend media which reached people at the maximum moment of concern, such as tube car panels.
Second, our participants mentioned that while the experiment was inconvenient for them, it was also a burden on their families. This led us to the insight that maybe it was better to encourage older, male sufferers to rectify the issue not for their benefit, but for their family's.
The cost of all this? About 50p on my phone bill.
Good insights can come from the simplest of places. Only if we recognise this will we undertake insight generation as frequently as we should.
Focus on the representativeness of the data
A seemingly simplistic point but the majority of online measurement collects short term effects only: sales, views or visits. This is a problem as what we measure shapes what we do. If we just measure the short-term then that is what we optimise to: more budgets go to the best performers, while the worst are struck off the plan.
And if we optimise just to short-term metrics then we under-perform in the long term. Binet and Field have showed in their analysis of the IPA Effectiveness database, that what works best in the short term isn't ideal in the long term.
Data alone are worthless, analysis is key
Data are nothing but raw material. They're a necessary condition for insight, but not a condition that necessarily creates insight.
Henri Poincaré, the 19th century French mathematician, pointed out the difference between science and facts. What he said is as relevant for insight and data:
"Science is built of facts the way a house is built of bricks: but an accumulation of facts is no more science than a pile of bricks is a house."
And just as facts require theory to become science, so do data require theory and analysis to become insight.
If agencies want a competitive advantage they need a combination of best in class data collection and interpretative skills. The best source of those interpretative skills is the study of human decisions making: behavioural science and social psychology. These tell us of biases like the survivorship bias but also hundreds of others that affect consumer decision makings.
Stop worrying about the changing man
The experience of Wald is useful. But there are plenty of other historical figures to learn from. We shouldn't ignore them because our technology has changed – fundamental human motivations are the same as they ever were.
As Bill Bernbach said:
"Human nature hasn't changed for a million years. It won't even change in the next million years. Only the superficial things have changed. It is fashionable to talk about the changing man. A communicator must be concerned with the unchanging man with his obsessive drive to survive, to be admired, to succeed, to love, to take care of his own."
The agencies that focus most on the unchanging man will win the insight war.