THE LONG AND THE SHORT OF IT needs THE WRONG AND THE SHIT OF IT

8

We’ve built a marketing culture that’s obsessed with aping winners and sweeps failure under the carpet. Our industry’s effectiveness bible, The Long And The Short Of It, is a huge compilation of lessons from winners, yet it’s in the losers that we can best understand success, writes Harry Guild, data strategist at BBH London

If you judge life by longevity, Dorothy is a winner. Of the 900,000 British babies born in 1913, she’s one of the 0.6% to make it to 100. She’s also a shitty advice-giver. For every Dorothy, there are millions of her generation who died young from smoking.

Survival from age 35 for regular smokers vs non-smokers among UK men born 1900-1930 (percentages alive at each decade of age):

Dorothy is a good illustration of how replicating a winner’s behaviour holds no guarantee of replicating their success. Indeed, in Dorothy’s case, it actively harms your chances. The truth is that success is often just down to luck. Sometimes people do all the wrong things and still end up winning. 

 

Despite this, our industry blindly clings to all winners, flukey or otherwise, as models for success:

This is known as survivorship bias: our tendency to construct faulty “learnings” from winners at the expense of more complex contextual datasets.

You may argue that WARC and the rest are analysing large quantities of winners, and so anomalies like Dorothy will be drowned out. Unfortunately, it’s not quite that simple. Often winners will share a common trait that is utterly unrelated to their ultimate achievement. Or it’s a trait that plenty of the losers have too, but nobody’s paying them any attention. For instance, if you look at the biggest entrepreneurs of the past few decades, one stat stands out: they’re mostly dropouts.

Steve Jobs, channelling his inner Dorothy, even claimed that the decision to “drop out and trust that it would all work out OK was one of the best [he] ever made”. But for the vast majority, dropping out of college is a disastrous decision. On average, a dropout is 52% more likely to be unemployed and will earn $9,350 less a year than their graduate peers. This dropout-graduate wage gap persists in every age group, even among those at the end of a forty year career – they never catch up from that initial setback. If you want to be as successful as the billionaire dropouts, don’t drop out.

“A stupid decision that works out well becomes a brilliant idea in hindsight… If you group success together and look for what makes them similar, the only real answer will be luck.” Daniel Kahneman, Thinking, Fast and Slow

This brings us to the effectiveness elephant in the room. In The Long and the Short of It, Les Binet and Peter Field analyse the relative success of 996 campaigns entered into the IPA Effectiveness Awards between 1980 and 2010. The crucial word is relative. Whether they drive a 50% increase in sales or 200%, every one of those 996 campaigns is a success in its own right. Nobody’s entering dross into the IPA.

As such, the study’s conclusions are only representative of the effectiveness of exceptional work. They cannot, as they commonly are, be complacently taken as a rule for all marketing.

Take the much-cited analysis of the relationship between campaign duration and effectiveness. Among this pool of winners, longer-term campaigns have the largest positive impact on price sensitivity, sales and profit. However, we cannot know that this holds true for non-IPA entrants. It could well be that long-term brand building campaigns are an all-or-nothing strategy; winners reap unparalleled rewards, but non-IPA entrants (ie. the majority of marketing) who go long may fare worse than other average ads who chase short-term salience instead.

HYPOTHETICAL: Relationship between Campaign Duration and Effectiveness in IPA Entrants and Non-entrants

Of course, this could all be nonsense. We just don’t know and that’s the point. The Long and the Short of It can tell us nothing about the relationship between duration and effectiveness in the average campaign. This is not to discredit an incredibly useful work, but rather to urge that it is read as what it is: a study of the successful. For a more complete understanding, we need a sequel that analyses 996 mediocre-to-poor campaigns. We need The Wrong and the Shit of It.

The power of studying both the winners and the losers is context. When failure is ignored, the difference between failure and success can become invisible.

Marriage is a good example of this. If you wanted to find the secret to matrimonial bliss, it’d seem reasonable to adopt the Binet and Field strategy of analysing a pool of happily married couples. You would find some patterns: happily married couples tend to have similar upbringings, shared ambitions and mutual respect. But you’d be missing a huge data-set of the unhappily married and divorced. And with them, you’d be missing the killer insight: happily married people started out happier.

Changes in life satisfaction in the years before and after marriage (0 years indicates year of marriage):

That one piece of data turns everything on its head. It suggests that the foundation of a happy marriage is built long before the wedding day and possibly before the couple even meet.

This insight only emerges from giving serious attention to the losers – an unfashionable thing to do in an industry obsessed with best practice guidelines and WARC 100s. The only losers our culture properly scrutinizes are those so spectacularly shit they spark an international backlash. But the overwhelming majority of bad ads aren’t Kendall clangers. They don’t inspire outrage, just apathy. They are blandly bad. They are lukewarm tap water, they are Muzak, they are meh. It’s these we need to open our eyes to.

Instead of looking to ‘best-practice’ that can only ever create me-too advertising, we need to have a much more rigorous point of view on the 84% of advertising that is not recalled or correctly attributed. Only through addressing these failures can the work improve.

It’s said that success has a hundred fathers, but failure is always an orphan. Be a better strat. Adopt some flops.   


Sources

Binet, Field, The Long and the Short of It: Balancing Short and Long-Term Marketing Strategies, IPA, (2013)  

Doll, Peto, Boreham, Sutherland, “Mortality in relation to smoking: 50 years’ observations on male British doctors”, BMJ (2004:328), pp.1519-33

Lucas, “Adaptation and the set-point model of subjective well-being: Does happiness change after major life events?”, Current Directions in Psychological Science, (2007:16), pp.75-79

Korn, “A bit of college can be worse than none at all”, WSJ, (2014)

Vital, “Entrepreneurs who dropped out”, Adioma, (2014)

That 84% stat – Based on a study by the Ehrenberg-Bass Institute, from a sample of 143 ads they found ad recall to be on average 30-40% and brand recall to be around 40%. In HBG Byron Sharp says ‘Up to 84% of all advertising spend is wasted’. Two other studies found broadly similar results: (1) one by Erik Du Plessis looked at recall for different lengths of TVC – finding it to be 20-40% (depending on 15–60s) (2) one by Ipsos found around 50% of the reason why TV ads fail to achieve impact is poor brand attribution.

8 Responses

  1. Great piece Harry.

    I’ve often pondered this question. One other potential example of survivorship bias that comes into play when agencies think about their work, particularly during the creative development process, is that most (all?) of the data is about campaigns that actually got made.

    Other than each individuals personal experiences of the near misses or ones that got away, we lose track of all of those ideas that could’ve happened.

  2. Such a great article. Should be regularly read by every marketing student, executive, etc. Loved it. Our cultural obsession on the “successful” and the exceptions has driven some bad behavior and decision making, from how VCs invest to how people make decisions and set expectations about their future. The drive to generalize certain behaviors like “reading a thousand books a year” (or whatever) and turn them into recipes for success is everywhere – and it’s not healthy. Anyway, loved this article. Thanks!

  3. Don’t the problems with ‘The Long and the Short of it’ run a little bit deeper? It is a very poor and partial data set, created for promotional purposes by agencies and clients trying to sell something. If you write a few of these papers, you get used to the extraordinary amount of wrangling and wrestling that goes into ‘constructing’ a case. If you work in one of the sectors of some of the award winning papers, you will often find the profit and ROI assumptions pretty laughable. So there are data hygiene problems at a paper by paper level, aggregated up to the big generalisations that are the stuff of the long and short of it. When you start looking at the correlations between the dodgy data sets of effectiveness papers and their interactions with the even dodgier data set of the Gunn Report, you are building a veritable tower of bullshit.

  4. Good article. Great, even. Reminds me of an argument against media personalisation based on ‘probabilistic’ mathematics vs. ‘falsification’ where the latter would, for example, show you recommendations on Amazon that it thinks you WON’T like and then uses the data to build recommendations of what you WILL. Probabilistic – you liked / other people with your traits liked this before so we think you’ll like the same thing or related thing again – creates echo chambers that spiral downwards. Falsification principle exposes you to new things, which is healthy in a democracy of information (and democracy generally)! Who says all shit is useless?

  5. hey, great argument. I have tried to make this same argument earlier too – https://truthaboutbranding.com/2017/11/13/effie-awards-propagating-wilful-ignorance/
    I think advertising agencies are not well placed to really figure out effectiveness. For a typical client, marketing is less and less important in comparison to issues such as technology, regulations and cultural shifts affecting profitability, operations and even survivability. Parameters of success are changing and affecting business model and yet, the idea of campaign effectiveness doesn’t move with the changing business dynamic. The effectiveness has to be seen in the bigger context of business , technological and societal changes.
    Things like IPA, effies etc are frankly a waste of good time and talent.

    Here’s something you might find useful though – 133 cases of startup failures. it taught me more about strategy than effie or IPA cases ever can. https://www.cbinsights.com/research/biggest-startup-failures/?utm_source=CB+Insights+Newsletter&utm_campaign=ca9eb4b2e9-Top_Research_Briefs_06_02_2018&utm_medium=email&utm_term=0_9dc0513989-ca9eb4b2e9-89866869

    cheers.

Leave a Reply

Your email address will not be published. Required fields are marked *