4 Things We Learned Analyzing Email Pop-Up Stats of 50 Online Retailers

13th Mar 2019 –
The same applies if you choose to add a delay before your pop-up:
If you’re still doubting these stats, here’s a real-life illustration at the website level.
The second row aggregates data from campaigns that did not include an offer. Their only pitch was a request to join an email list, subscribe to the website’s newsletter or receive exclusive offers.
All in all, timing is a double-edged sword. If you want to focus on the number of emails you can collect, then it’s best to display your pop-up sooner. If you want to preserve the UX, it’s better to wait. Personally, I lean towards the second option.
Offers do work. But what kind of offers? This A/B test gives us an additional hint.

Offers Matter (a Lot)

The results of this test show that:

How Much Do Offers Matter?

When setting up overlays on a website, one of the most difficult questions to answer right off the bat is: When should I display the pop-up?
That’s why I felt lucky to have access to the pop-up stats of 50 top e-commerce retailers. This gave me the chance to review real-life data and learn the secrets of effective pop-ups.
In this article, I’ll share our findings. Here are the results of an AB test run by a high-end audio system retailer (I took this one as an example, but we’ve reproduced these results more than once):

Do Offers Always Matter?

Does this mean that the best email pop-ups are displayed as soon as a user opens a web page?
But again, it turns out the copy is worth testing. In this section, I’m going to share an A/B test that changed my mind about the importance of wording for pop-ups.
Until now, we’ve reviewed pop-ups’ fundamental elements: the offer, the targeting, and the adaptation to mobile devices.

First, experiments can have a huge impact. Most of the tests mentioned had a dramatic impact on conversions and UX.

What Kind of Offers Work?

While the second focuses on the number of watches you could win.

Secondly, you have to find the delicate balance between UX and immediate conversions to achieve your best opt-in pop-up results. A smarter pop-up doesn’t always collect more emails. But it will certainly annoy fewer people.

When the sweepstakes are up, their pop-up can convert up to 23% of their visitors into email subscribers (!!). When the offer is removed, their conversion rate goes down to less than 2%.

This second version is more vague. It mentions “offers” without detailing what’s behind this wording. It had a conversion rate of 5.8% (a 31% gap with version 1).

In August 2016, Google declared war on “instrusive interstitials.”

Timing Is a Double-Edged Sword

This skateboard shoe retailer alternates between a pop-up inviting her visitors to join its email list and a pop-up offering a chance to win a free pair of shoes.

Why Displaying a Pop-Up on Landing Can Be a Good Idea

Last year, we reviewed 300 ecommerce websites. 76% of the stores using pop-ups were displaying them as soon as the visitors landed on the website.
The first version insists on the total amount.
Webmasters had no choice but to adapt their campaigns to the new policy. It meant either creating a mobile campaign that displays well on a desktop or separating campaigns altogether.
Sometimes, it’s worth sacrificing conversions if it means that your users have a better experience on your website.
This client tested two versions of his campaign. The first version mentions a coupon. It had a conversion rate of 7.6%.

Here our client, an adult product retailer, tested a 10-second and a 20-second delay.

We thought this change would benefit all sides: from visitors who would be happier about their navigation experience to marketers who would collect more leads with improved forms.

The first version where the pop-up is displayed right away has a lower conversion rate.
But, it ended up collecting 173% more emails than the version displayed after 2 pages.

These pop-up stats taught us a few things.

And in this case, I would not recommend risking your SEO traffic for the sake of collecting a few more emails.

As you can see:
In other words, I usually don’t focus too much on the copy of my pop-up campaigns.

Should All Pop-Ups Be Displayed Right Away?

We also assume that pop-ups displayed on landing work better if the user already knows the brand or has visited the website before. Since she already knows what she can find on the website, she’s more eager to share her email right away if it can get her a good deal than if she didn’t know what she would find on the store. Unfortunately, we’ve never had a chance to test this hypothesis so far.
The first takeaway of our study is that a significant proportion of retailers haven’t adapted their strategy to these new guidelines. 17 out of the 50 retailers that we reviewed are still displaying the exact same non-mobile-friendly pop-ups on mobiles and desktops.
2) In general, all the details that make a pop-up matter are worth testing: from the targeting, to the offer, to the design and the wording that you use.

(Although to be honest, I’m still not convinced that the color of a CTA alone can have a huge impact).

Creating Mobile-Friendly Pop-Ups Won’t Help

Google Guidelines for Mobile Pop-Ups

It turns out, things didn’t really go that way.
The upper row aggregates the data of pop-ups that included an offer. By offer, we mean everything that might convince the visitor to subscribe (from a coupon, to a chance to win something, to an ebook, etc.).

This watch store tested two headlines. As you can see, the two mention exactly the same prize: 20 watches, worth 00 in total.

The Impact of the Guidelines on Conversion Rate

Having a higher conversion rate means that you’re annoying a smaller percentage of your users. In the first AB test we mentioned, the difference between version 1 and version 2 is that only 92% of the visitors who saw the second version chose to close it, while more than 96% of the users who saw the second version chose to dismiss it. Put differently, the second version annoyed two times fewer users.
E-mail pop-ups are one such case in point where it’s difficult to find guidance. The available data about their performance is scarce. Case studies are often promotional and look biased. And to my knowledge, it’s difficult to find the results of A/B tests involving overlays and modals online.

To be honest, I’m the kind of guy who likes to focus on the main elements of a marketing strategy. I’ve always felt uncomfortable when someone brought up the idea of testing the impact of using a different picture to illustrate a home page, checking the impact of font types or A/B testing a wording.

What Lesson Can We Draw from These Stats?

Again, the conversion of the version that’s displayed later is slightly better. But it ends up collecting -9% fewer emails.
We were quite shocked by this statistic.

As marketers and product managers, we often have to make decisions without data. We have to trust our gut, try something new and…sometimes fail.

The era of intrusive email modals is not over yet (a pop-up on AmericanApparel.com)

Don’t Forget to Optimize Your Copy

And it looks like it doesn’t impact their pop-ups’ conversion rate. These retailers have an average conversion rate of 4.41%, 27% higher than the stats of their counterparts who chose to comply.
Get ready for a whole lot of stats and A/B tests’ results!
All things being equal, the first headline (13.40% conversion rate) achieved 2x the subscription rate of the second variant (6.70%).

We mentioned stats. So let’s start with metrics about the impact of offers on the performance of opt-in pop-ups.

Here are the results for our sample.

Hell, no!

So not only are potential subscribers looking for offers – it looks like they prefer to know exactly what the offer is. The more specific you can be about your offer, the better.

Why? Because the longer you wait, the fewer people will see your campaign. And even if the subscription rate is higher, it doesn’t compensate the smaller size of your sample.
Let’s review one last example.
Well, stats reveal that this strategy is not as dumb as it looks.
When looking at these results, it’s no wonder why a majority of the retailers in our sample (42 out of 50 retailers) included an offer in their pop-up. The retailers that include an offer have a conversion rate that’s almost 2 times higher than those who don’t.

Before You Invest in Pop-Ups

Indeed, as a visitor or a CRO specialist, it could sound weird. Why bombard your visitors with a modal right away? It doesn’t give them a chance to learn about your brand, services, etc.
1) The words you choose to describe your offer or content do matter. And they can have a huge impact.