I Don’t have Enough Traffic to A/B Test. Now What?

25th Apr 2019 –
And that’s what’s important — because once you shift your focus to learning by trying rather than accepting the norm, you’ll start developing the culture of experimentation, a prerequisite for effective CRO.
The thing is that you can start testing ad copies even on a limited budget.
For instance, email marketing solution provider ConstantContact recommends a higher sample size at twice the minimum size that HubSpot allows. It suggests that for meaningful A/B testing, the sample size must be at least 1,000 contacts. So, if you’ve 2000 subscribers, you should be good to test (with your A and B versions each getting 50% of the total contacts as test traffic).
It depends a lot on the platform and the industry (among other factors), but there’s data to help you do the budgeting.
MailChimp also recommends somewhat larger lists to run A/B tests. It suggests that you must have at least 5,000 contacts to run tests. But because so many small business contact bases are much smaller than that, it recommends testing ideas across the full list:
Here are 150+ email split testing ideas for your email campaigns.
Social media is another channel that lets you test your ideas, no matter how small your social following might be. Besides, in a way, on social media, nothing actually limits your reach.
For example, AdStage analyzed over 110 million ad impressions (on Google Adwords) in Q1 2018 and found that on an average, you need 6.91 to get a thousand impressions (CPM) on Google Ads. Last year, it was about 13% cheaper.
Data from Statista research states that the average CPM (cost-per-mille or cost per 1,000 impressions) for Facebook is USD 1.26. Other Facebook CPM average cost estimates from AdExpresso and  WordStream fall in the -10 range.
This kind of explains why CRO isn’t a million customer kind of market as there are just a few 100k websites (or businesses) with the footfall to test with any success and the budgets for the pricey (rapidly going “enterprise”) optimization solutions!

Should Low-Traffic Sites Rule Out Testing?

Given that an average small business only gets a few hundred visitors a day to its website, A/B test experiments don’t make sense for them.

Also, here’s a quick guide on email A/B testing.
But:
About 93% of companies see better conversions with personalization.
Let’s begin!
And you can start with personalization right away.
On your product page for your customers, you could show content such as product guides and videos, quick links to access support and more, so that they get a more rewarding product experience. Whereas the version you show your leads — for example, those who signed up for a trial — could focus on your product’s features or case studies to help them convert.
Surprisingly, email marketing is one channel that lets you experiment with different ideas, without needing tons of subscribers (or “contacts”).

Testing with Email

Says every email marketing software provider right from MailChimp, ConstantContact, and Campaign Monitor to HubSpot (that supports email marketing in its marketing automation software).
So if you want to run an A/B test using HubSpot’s email platform, all you’ll need are 1,000 contacts: 1,000 contacts is also our benchmark for running A/B tests on samples of email sends. — HubSpot
You can run a host of social media A/B tests to uncover such trends and boost your social media performance. We’ve published a great primer on social media A/B testing here. Read it to learn how you can start with social media A/B testing, the different social media A/B tests you can run, and the top social media A/B testing best practices that you must keep in mind. 
We know there is a ton of debate raging around statistical significance, the 95% mythical (or not?) mark, testing and traffic, and more.
The best part about personalization is that even low-traffic websites can work with them as the traffic constraints that apply to CRO tests don’t apply to personalization.
That’s for sure.
So how much do a few thousand ad impressions cost?
It doesn’t matter what growth stage you’re at, you can invest in developing a culture of testing. It will come in handy when you actually have enough traffic to your website to go for hardcore, definitive CRO experiments.
For example, if you own a small business, running experiments will help you move away from gut-based toward data-driven decision-making. It will force you to look at data … and with in-your-face numbers, you’ll stop defaulting to *I think* strategies. This data-backed approach alone will transform how you build and grow your business.
Whether it’s Google Adwords or Facebook ads, most paid networks support experiments so you can get the highest ROI.
Because to derive statistically significant insights, you need to show your experiment’s variants to (at least) thousands of users.
Personalization or personalizing a visitor’s website experience based on what you know about them is another common form of experimentation.
Let’s start with HubSpot.
Plus, with a tool like Convert Experiences, you can set up personalized website experiences within minutes. Just set the targeting rules and you’ll be able to show personalized messaging to your different audience segments — prompting each segment to move to the next stage of their buying journey.
Also, we suggest recording around a 1000 conversions to yield conclusive evidence of the superiority of the control or variant.
Now, you may be thinking, “My list size is only 500 subscribers, how do you expect me to run a successful test?” My answer is that you’ll need to set your sights on a larger MDE [Minimum Detectable Effect (MDE)]. When you increase the MDE, your required sample size decreases. So instead of needing 592 subscribers per variation to detect a 20% relative effect, you would only need 94 subscribers per variation to detect a 50% relative effect. Note that the baseline conversion rate also plays a part in your sample size – the lower your conversion rate (in this case, open %), the more subscribers you’ll need to run an accurate test. — Campaign Monitor And there are many benefits to running them.
Campaign Monitor also says that you can run A/B tests even with small emails lists. Here’s the math behind it:
The reason behind this is that with personalization, the end goal isn’t to optimize conversion rates. Instead, its goal is to offer a more relevant user experience that leads to more conversions or higher customer satisfaction and retention.

While low website traffic can restrict you from running substantive CRO experiments, it can’t stop you from “learning.”
So if you’ve about 500 or 1000 subscribers, you’re good to test, provided you have a responsive database that is interested in what you have to say.
“Says who?”

A/B Testing with Paid Ads (PPC)

Testing doesn’t work for low-traffic websites.
Running experiments will also teach you to learn by trying, rather than accepting the status quo. It will inspire you to see if the principles-believed-to-work actually work for you.
In fact, there are many experiments that even a small business can run.
And her tests bring some great wins for us. For instance, her experiment using a certain set of hashtags once won us a retweet from Google Analytics!
That doesn’t mean a small business can’t run experiments at all.
For subject line testing, we recommend testing each line on a sample size of at least 1,000 contacts if possible. 1,000 users for each version is typically a good benchmark for any A/B test, but if you don’t have enough contacts or visitors to hit those numbers, it’s still worth running the test. — ConstantContact
While it’s true that having a huge sample size will help you reach statistical significance much sooner (and with much more confidence), but you can start testing with your small list as well!
Trigger Warning:
If you are an optimizer who feels strongly about this stuff, you might get triggered. Well, the comments are open and we can have a really enlightening conversation.

An (Almost) “Traffic-Proof” Way of Experimenting: Personalization

You might say that it’s easy to dismiss quick wins as (maybe) flukes BUT when you see a set of hashtags routinely earn more exposure than the others you test it against, you’ve more than a test result … what you have is a trend.
So for example, at Convert Experiences, even we don’t have tens of thousands of followers, yet our awesome Social Seller keeps experimenting with elements of our social media copy such as its hashtags, images, and more.
But to get statistically significant results from your PPC experiments, you need a few thousand impressions of each ad copy. Google recommends testing ad groups that get at least 5000 impressions/week.
We recommend you send combinations to at least 5,000 subscribed contacts to get the most useful data from your test. If the size of your list or segment doesn’t allow for 5,000 subscribers in each combination, consider testing your entire list and use the campaign results to inform future campaign content decisions. — MailChimp
Now, that’s some social media exposure, isn’t it?
So start running whatever experiments you can over whatever marketing channels you use. Here are some of your options.
Social media networks like Facebook also offer accessible CPM options.
For example, you could create different versions of your product page to show to your customers and non-customers (or leads).

Wrapping it up …

Finally, running experiments will prepare you for some “real” CRO when you have “enough” traffic as it helps develop the right mindset for testing — a prerequisite for effective CRO.
However, a few email marketing providers recommend a slightly higher contact base.
In fact, at Convert Experiences, we often recommend our users run A/B tests — which are the simplest CRO experiments — only if they can send about 10K visitors to each variant.