Experimentation Works (Stefan Thomke) – good for marketing and customer experience leaders?

Does it sometimes feel like you’ve tried ‘everything’ smart in your marketing – and the only answer seems to be to spend more?
That’s fine if you have infinite money (and infinite time) but wouldn’t it be great to out-think the competition instead of trying to out-spend them? Experimentation Works by Stefan Thomke might be the answer.

(Where the Corner Table Business Book Club meets: mentally if not physically at the moment)

I think Stefan Thomke’s new book might have a lot to offer everyone in marketing and customer experience and anyone that wants to grow their business.

It’s a radical take on how we should run our daily lives. I’ve read it and I’ve already started putting the thinking into practice. Even better, I met a co-founder tech client for lunch the other day and he’s basing everything he does in his new business on the book!

So what’s it about?

One of the joys of my job is spending one evening each quarter running a business book club for a few close friends who are leading marketing and customer experience departments.

We pick 3 non-fiction books each quarter plus one fiction book and then meet up (virtually) to discuss them.

There’s nothing like a deadline to motivate me to get on and read the books I know I should.

But even better than that, is having my thoughts challenged by smart people from the marketing and customer experience worlds.

Experimentation Works Stefan Thomke certainly did that.

Below are the notes I made during the Business Book Club: if you’d like to find out more, I loved the book and would be happy to have a 20-minute chat about it with you if we can both find the time.

(If you’d like to join a Business Book Club with me, email me please and let me know. I think we’re full at the moment but we can talk about what’s coming up.)

So here are the notes that I think are most relevant for marketing and customer experience leaders about Experimentation Works

Synopsis of Experimentation Works (Stefan Thomke) for marketing and customer experience leaders

1. EXPERIENCE ISN’T ALWAYS THE GREATEST GUIDE

• The new (2011) CEO of J. C. Penny tried to recreate his Apple Store success – and nearly bankrupted the business.
• “The iPhone will never appeal to business users because it doesn’t have a keyboard” [Steve Ballmer, Microsoft CEO, 2007].
• A Microsoft employee’s idea for how Bing displays ad headlines was dismissed for 6 months as a low priority, but was finally A/B tested. It produced a 12% increase in ad revenue.
• Human bias tends to resist changing long-held beliefs or practices.

2. THE SOLUTION ISN’T TO TEST OCCASIONALLY AND LOOK FOR SUCCESS:
it’s to test often + early + fast, then treat the expected high rates of ‘failure’ as information

Jeff Bezos: “Failure and Invention are inseparable”
Booking.com is testing more than 1000 hypotheses each day
Google has about 24,000 algorithm tests/year- which led to ‘only’ 516 changes
He provides ‘evidence’ to support the impact of experimentation:

(and while Steffan Thomke is good enough to accept that ‘correlation isn’t causation’, he’s quick to say why this chart proves his premise. I think this is atrocious ‘science’!)

IF WE ACCEPT HIS PREMISE, WHAT CAN WE LEARN?

1. There are ‘ground rules’ that ensure a successful experiment:

• the experiment is designed to test a specific point (‘hypothesis driven’), rather than “Let’s do this and see what happens”
• stakeholders are committed to abide by the results
• the experiment must be doable – the variables are controllable
• the results must be reliable (I’d say, “scientific”)
• there’s a way to understand cause and effect – the British Navy thought the acidity of citrus fruits prevented scurvy
• there’s a way of harvesting maximum value from the results
• (strategic decisions are driven by experimentation)

2. Leaders can create a ‘high velocity’ testing culture:

• Create a Learning Mindset – ‘failure’ is desirable (provided its early enough and cheap enough)
• Incentivize experimentation –
• Encourage humility and self-awareness (note D2 below about human biases)
• Insist on ethical integrity – p.136 Facebook’s experiment and outrage
• Trust the tools
• Balance ‘exploration and exploitation’ (see note D4 below)
• Adopt a newer leadership model (see Note D5 below)

3. Leaders can use brand language as a secret weapon when testing:

• Remember what President Obama’s strategist said: “About halfway through the campaign we figured out that of all variables that affect user behaviour (design, usability, imagery, etc) copy has the highest return”
• Brand language is the easiest, fastest, cheapest thing you can experiment with.
• Why don’t more people get outside help in experimenting with their copywriting – in-house writers are focused (blinkered?) …what would happen if you brought in some help from outside…just as an experiment.

My Thoughts, questions and prompts for you if you’re reading Experimentation Works by Stefan Thomke

1. Will offline be as testable as online? – Thomke devotes a whole chapter to testing online – which is a ‘pure’ space to test in. Offline, real life is messy. So, how do you test? (Image above is from the movie about how Ray Kroc of McDonald’s did it in ‘The Founder’. And Steve Jobs built a life-size replica of the first Apple Store.)
   So, what do we do already? What else can we do? How bold would you be in testing something? And isn’t every business about to start interacting with its customers digitally….!
2. Me, Human – Human biases certainly compromise analysis
   Seeing relationships when something is random;
   Choosing results which confirm our bias.
   We reject evidence which conflicts with practice/belief/pride]. But humans are successful because of highly intuitive statistical averaging ability. What do you need as a reliable enough premise to create a hypothesis and test it?
   How do you avoid ‘p-hacking’ which is current in scientific community – where you choose which data to include, in order to give greater statistical credibility to your results.
3. Me, Machine – How much could you take humans out of the loop for deciding what should be tested? Human bias means we’re only looking for the things we’re looking for, and are reluctant to consider other things. P. 109: The Google employee who quit because he was asked to prove his opinion on whether a border should be 3, 4 or 5 pixels wide. He was wrong, but so were management in paying him to be in that conversation.
4. There’s an essential tension between efficient incremental improvement Vs outsized, non-linear innovation. How do you balance this?
   He cites 3M where an incoming CEO (James McNerney) tried to apply the Six Sigma methodology, originally designed to remove variability in manufacturing, to their innovation process without success (p115)
   But his successor, George Buckley, freed scientists from Six Sigma saying “That’s now how creativity works”. P. 145
   Well, I think it is, in some way. Control the environment, the goals, etc and then create more. Ask me about my experience as a copywriter at 2 vastly different (but both highly successful) agencies: BBH Vs Legas Delaney.And Jeff Bezos’ quote: Wandering in business is not efficient but it’s also not random”
   Do you hire incrementalists and wanderers? This seems to be the key question in Experimentation Works Stefan Thomke’s book.
   When would you prioritise innovation over experimentation?
   Are there industries where one is more important than the other (fashion’s twice yearly innovation doesn’t allow for experimental testing)

5. What’s the role of the leader in an experimentation culture? It’s the same as ever: To create goals, systems and cultures.

Final questions as always:

What would you take from this book and use?
And did you enjoy the book?

Questions? Want to talk about the book? Or about whether you could experiment better with your brand language?
Drop me a line and let’s see if we can find 20 minutes to talk.