25th Jan 2019 – 
Human behaviour is so complex that we’ll often overlook the far ranging effects our experiments can have. Because when we’re unaware of it, we’re not analysing it. And if we don’t analyse it, we could be pushing tests that hurt people’s wealth, health and happiness.
Well, that’s where the waters get a little murky. A/B testing is all about growth, we both know this. And as such, it’s easy to overlook the effect our tests have on the end user. We probably just want to grow our revenue and make a profit.
The fact is, businesses drive more revenue, customer lifetime value and margins and they do it quicker when they experiment.
I was recently speaking with Bart Schutz, a behavioral psychologist and an expert in A/B testing. On our call he gave the perfect example of an A/B test that on the surface looks like a winner but had huge ethical implications for the end user.
And how without proper data analysis and research by psychologists, this link might never have been found.
This a very specific example and might not relate to your business at all, but it does show how A/B tests can hugely impact the end user.
More and more companies are seeing the importance of asking themselves the ethical questions and aligning business interests with the interests and happiness of their users and customers.
That was until their lead psychologist found an article that researched the association between cleanliness and feelings of safety, especially among women. And by emphasising the cleanliness of hostels, they could induce implicit feelings of safety, possibly driving more women to what could be actually unsafe hostels.
How Do Ethics Fit In with A/B Testing?
The next time you make an A/B test, ask yourself the ethical question. Right now, with all the trustworthy data and knowledge you have, could this test have negative implications for your end user? And if you choose to experiment, make sure company interests and user interests are aligned.
Let’s wrap this up with some action items you can work on right away. What can you do to make sure you’re being ethical with your testing and are aligned with your customers’ best interests? Here are some points that can give you better direction:
The next step is making sure that our experiments and result analyses are done accurately.
And It’s never the tactics that are unethical, but the intention and the cost/benefit analysis for those who are ultimately subjected to the testing.
Certainly It’s never black and white, especially in the murky waters of ethical A/B testing and driving business growth.
We may even try to stay away from the things we believe are frowned upon, like countdown timers that aren’t real, or stock counts that aren’t linked to the back end.
So let me ask you a question, how are your business metrics taking your customers best interests into account? And if not, how can you make sure they do moving forward?
So What Does Ethical A/B Testing Look Like?
However after doing the experient and analysing the data, it turned out to create a big ethical dilemma. What was in the businesses best interest wasn’t necessarily in the customers, especially their female customers, best interest.
Furthermore, when we take into account the end user and base some of our metrics on whether a test is good or bad for them, we help create businesses that do good in the world and stay around for the long term.
Although this is a tough one to do. We must weight up short term gains with long term KPI’s, taking our end user into account.
Being experiment driven can also save a lot of time and money validating ways to best prioritize IT resources based on the value they bring
Have you ever thought about ethics in A/B testing?
What Should You Consider When Approaching A/B Testing?
We can guess all we want about the predicted results of our A/B test experiments.
And the only real thing we can do to be ethical, is to be rigorous in our analysis of the possible negative or harmful effects of our testing.
The fact is, we first need trustworthy data – based on proper research, with large data sets.
But until we’ve conducted the experiment, got back accurate data, and done the right analysis… we simply don’t know the ethical implications.
The first step is to ensure that our data is completely trustworthy, then start experimenting and analysing based on this trustworthy data, ensuring trustworthy outcomes.
On the surface this A/B test looked to benefit both the business and the customer. The business benefited by an increase in bookings, and the customer seemed to benefit by seeing the cleanliness of the room.
Imagine going to a booking website and seeing the cleanliness of hostels emphasised. You’d probably think it was a great idea. As you can imagine, the hostels with the best cleanliness score would get the most clicks. And they did. Sales increased, everyone was happy, and with no ethical issues in sight.
The Future Of Ethics And User Interest As A Business Metric
Sure sometimes it’s obvious, and if it is, then it’s our duty as humans to speak up and let our voices be heard. Whether that’s from the client, the agency, or whoever is doing the testing. But as we’ve seen from the booking site example, sometimes the ethical implications of experiments are very hard to see.
Sure, we don’t set out to do anything unethical.
As end users ourselves, we’ll always gravitate towards the businesses that have our best interests at heart. And it’s these businesses that become brands which flourish in the modern world of increased accountability and transparency.
But what if our product genuinely can change people’s lives? What if our product can make people happier, healthier, or wealthier? We can argue that if our product does do one of these, we should do anything in our power – use any marketing or sales tactic – to get our product into the hands of the end user.
We all have our own ethical compass guiding us to what’s right and wrong. And as digital businesses, we should have our user’s best interests at heart, right?
And that’s really the message here.
So let’s take a deeper look at how we can navigate these ethical waters, keep our customers happy, and grow our revenue.
- In an ideal situation you’d have behavioural psychologists looking through research -analysing and identifying correlations in the data. But if this isn’t possible, then really dive deep into the data and the possible effects of the experiment on your end users to the best of your ability. This is how you’ll catch tests that on the surface appear to be ethical but actually may cause harm.
- Ask yourself ethical questions before the test begins and encourage team members to speak out if they have issues with testing. We’re all human and have differing levels of ethical sensibility. So what crosses the line for one person may not for another. That’s why an environment of open discussion and freedom to speak out can be key.
- Guessing and opinion can only take us so far. The only way to know the result of an a/b test and it’s ethical implications is to run the experiment and gather accurate data. In the words of Nike, “Just do it.”
- Sure we can ask our users to give permission in some way to the testing, or even ask their opinion on the tests and whether they feel manipulated by them. But keep in mind that how people view ethics in relation to commerce can vary wildly from country to country.
- Look at your KPI’s and see how you can align business goals with the health, wealth, and happiness of your end users. ‘Time spent on site’ is a good metric for media companies wanting to monetize their platform. But ‘time well spent’ is a much better way to look at monetization in the context of the experience end users have with your brand. Users who are happy and healthy and derive value from your products in the long term will always lead to an increased lifetime value and reduced churn.
We just need to make sure that our success metrics include both business and user interests. And we should always make an effort to research and analyse the ethical implications of our actions.
Like I said, it’s murky waters we’re swimming in. Business KPI’s and user interests are not always aligned. In fact, I’d hazard a guess to say that they’re definitely not aligned for the majority of companies.
