The Customer Experience & Experimentation Dinner took place on Wednesday 23 May in the lofty heights of AquaShard. Over fine food and wine, and the heads of London's commuters, industry leaders discussed experimentation as a means of achieving growth.

Members were invited to discuss:

• How brands test, validate, and learn across all channels and applications – without risking hard-won customer loyalty
• What are the cultural and organisational obstacles to embedding a culture of experimentation? How can these be overcome?
• What are the most common reasons why digital initiatives sometimes miss the mark?
• Which area of digital experience are ripe for experimentation?

The Customer Experience & Experimentation Dinner was conducted in partnership with Optimizely. Optimizely is a leading experimentation platform, which enables business to deliver continuous experimentation and personalisation across channels.

Experimentation Practises

Data-based business strategies are clearly a great bulwark against avoidable business catastrophes; experimentation can be one of the great tools to access strong data sets. But where should you start?

When it comes to experimentation the best advice is to “just get started”; start small, and make sure you’re testing the right thing: prioritising business changes. The easiest tests are changes in “verbage”, which includes testing things like ‘buy now’ versus ‘pay now’. However, such tests are simplistic and business should only be doing these at the start of their experimentation journey.

Secondly, businesses should be sure to build out usable analytics, as data is ripe with ideas for further experimentation. No analytics, no (good) ideas. Third; build out a sophisticated customer journey model. Ask yourself, "Where is the customer journey falling?". This is the best place to test.

Once they are confident, businesses can start to experiment cutting or adding length to the customer journey. Start small; but more importantly, start. If you bake experimentation into your process, put it into what you do, it will be what you become.

Finally, remember more variables and more lines of code “win”. Counter-intuitively simple experiments just aren’t as clear or dramatic as a multi-variable test, and you’ll realise you can really flex your creative muscles when you have more variables and more information.

Good Segmentation

Data is powerful, but it was agreed how you segment and think about that data can be what makes your data the new gold or the new pyrite.

With today’s demanding consumers and shortened attention span, one often finds that with the slightest variation on what a customer is looking for they can begin to de-engage. Getting rid of all unnecessary steps, whilst keeping them available for customers who would prefer the extra service, is the best policy. Good personalisation and segmentation differentiates those companies that delight from those which are left behind.

The most common mistake in segmentation is to segment by general demographics like sex, age, or nationality. You must segment by behaviour; track people by what they do, not by who they are.

The advantage of using real behavioural data is the discovery that you can better assess risk and then better reduce it. Data around where someone purchases goods, when they pay their phone or electricity bill, can give you better insights than someone’s credit score. With big data and paying behaviour patterns people who appeared like a risk, suddenly are not.

To really supercharge insights, the real power from segmentation and behavioural requires aggregated data across multiple factors. If you combine card, carpark, and utility; you have a dataset which is more powerful than the sum of its parts. You can identify exactly what kind of customers are present in a prospective business location, what effect your competitors will have on that customer set and discover more insights that just boggle this editor’s mind.

Turn away from the executives and toward behavioural data

Fear the executive.

Too many seemingly common-sense ideas are pushed through without testing, or even ignoring, data-based tests.

An example:

There's a service provided to a customer; but there are breaks in that service, during which the customer grows impatient. An obvious executive solution appeared to be to send over a quasi-solution to tide customers over, during the break. This didn't account for the realities of consumer psychology: customers hated it. It was a half-arsed solution which smacked of unprofessionalism and lack of care. People are more inclined to understand a full break, as long as there’s a full solution in view.

Anyone with a basic grasp of popular behavioural economics theory is aware that seemingly logical strategies just can’t keep up with the wild illogicality of human behaviour. It's essential to test, rather than intuit.

Years ago, a games company tested a page selling two products: the ‘Regular’ versus the more expensive ‘Deluxe’ version of the game. There was only one variant: whether to include a banner that promised an additional 20% off your next purchase. Intuitively, this should have been another draw - it sounds like a great value-add.

Remarkably the site with this extra offer had 43% lower uplift compared to the control. And the reason for this can be found in behavioural economics. As the psychologist Daniel Kahneman discovered, humans’ loss aversion isn’t proportional to gain satisfaction; we are more averse to potential loss than motivated by potential gain. The reason the offer lowered uplift was that the 20% was perceived as a loss. They felt like they were somehow losing out.

Furthermore, they had made the calculation more complex: now they were thinking about some discount for some other game, which they weren't sure they wanted yet.

Behavioural economics can help prepare us for wild anomalies; but the theory has one central principle over others. Human behaviour is too chaotic to predict. Observation is needed. Which means experimentation, and scientific tests.