Recently, we were asked to help an ailing brand. After dominating its category for decades alongside a very similar competitor, recently the brand lost the top spot to its rival. What had gone wrong?

We immediately noticed something striking. Yes, our brand had lost market share to its doppelgänger. But more surprisingly, for years, both brands had been losing market share to a host of smaller competitors which now accounted for a bigger share of the market than either 'market leader'.

Further analysis suggested an explanation. The two big brands had followed identical marketing strategies. In an effort to increase RoI and efficiency, each had reduced marketing expenditure. Each had cut emotional brand advertising in favour of harder selling stuff focused on 'new news'. Each had replaced expensive broadcast media with cheaper digital channels, tighter targeting allowing both brands to reduce 'wastage'.

But without big, famous advertising, the public started to forget about our two brand leaders. And when they did think about them, they felt less warm towards them. So despite product improvements, ratings deteriorated and people began experimenting with alternatives.

Surprisingly, our clients hadn't noticed. They were so focused on their immediate rival that they'd failed to spot the little brands stealing their customers; we suspect the other brand had the same blinkered view.

We've seen this phenomenon before. Adam Morgan even has a name for it: 'The Mephisto Waltz'. Two big brands become so obsessed with competing against each other that they become mirror images – each copying the other's strategy, each benchmarking itself against the other, and making it all too easy for challenger brands to sneak in and grab share. If the market leaders don't notice in time, then the Mephisto Waltz becomes a death spiral.

But how does this happen? Why don't brands realise what is happening? There are three main reasons.

First, a focus on efficiency, rather than effectiveness. Big brands with high market share find it hard to increase revenue, so they tend to focus on cutting costs. Our brand was relentlessly focused on short-term RoI, and its competitor seems to have had the same obsession. But cutting their budgets destroyed the foundations of their former success.

When marketing efficiency is the main concern, targeting and segmentation come to the fore, and that's the second problem. Like so many well-trained marketers, our clients focused on a tightly defined market segment. When we showed them how their customers were defecting to a host of smaller brands, they argued that these were not competitors. Some were too cheap, others too premium. Clearly they defined their market segment so narrowly that there was only one other brand in it - their big rival.

But as Ehrenberg and Sharp have shown, markets are much less segmented than people tend to believe. We use a repertoire of brands - a surprising number of 'upmarket' Waitrose shoppers also pop into Lidl occasionally - so all category brands compete with one another to some extent.

The third problem is short-termism. When brands narrow their focus and cut brand investment, the result is a long, slow decline over years. Our clients were so ruthlessly focused on month-to-month sales fluctuations versus their big rival that they never noticed the long-term decline.

The lesson? In marketing, remember to waltz with more than one partner!