Comparing Email Subscribers Acquired from Pop-Ups vs. Other Methods
In August I published a post examining a common question in email marketing – do email subscription pop-ups work? The post detailed how I went about implementing pop-ups via OptinMonster (affiliate), as well as the results I attained in using them on two separate sites. Despite “most people” claiming they hate them, the results were clear: email list subscription pop-ups work – and they work well.
I received an interesting follow-up question from one of my list subscribers in an email:
“Have you found any difference in the quality and engagement of pop-up subscribers vs. subscribers obtained through other methods?”
It was a question I didn't have a data-based answer for at the time. But it's been almost six months since I wrote that post and now I have some long-term data to look at when comparing the behavior of subscribers obtained via the pop-up acquisition method vs. other methods.
I took a look at the stats for my Sound Bites newsletter from September 1st, 2015 through January 15th, 2016. My subscription rates have maintained the increases in signups, confirmations and conversion rates over time that they experienced with the initial use of the popup:
*Note: The photo for this article is edited. No information shown in the photo is altered – some of the statistical information is merely removed, and a blank space appears in its place.
But what about the quality of those signups?
Here's what I found:
Confirmed subscribers:
- Percentage of non pop-up subscribers since 9/1/15 that confirmed their subscription: 88.1%
- Percentage of pop-up subscribers since 9/1/15 that confirmed their subscription: 84.45%
Follow up message open rates:
- Open rate of my follow-up message to confirmed non pop-up subscribers: 69.61%
- Open rate of my follow-up message to confirmed pop-up subscribers: 68.66%
Unsubscribe rates:
- Percentage of confirmed non pop-up subscribers since 9/1/15 that have unsubscribed: 5.19%
- Percentage of confirmed pop-up subscribers since 9/1/15 that have unsubscribed: 7.46%
No opens rates:
- Percentage of confirmed non pop-up subscribers since 9/1/15 that have never opened an email from me: 19.22%
- Percentage of confirmed pop-up subscribers since 9/1/15 that have never opened an email from me: 18.4%
Last week's open rates for Sound Bites:
- Percentage of confirmed non pop-up subscribers since 9/1/15 that opened last week's Sound Bites: 28.31%
- Percentage of confirmed pop-up subscribers since 9/1/15 that opened last week's Sound Bites: 28.36%
My open rates for both pop-up subscribers and subscribers via other methods are both higher than my industry averages according to this data from Mail Chimp and these stats published by ConstantContact.
Comparing Conversions
Comparing the percentage of subscribers who click on links within my newsletter is a lot trickier to do. Aweber, unfortunately, doesn't allow you to segment subscriber data by “clicked any link” and instead makes you choose to segment subscribers that have clicked on clicking individual links in specific newsletters. So, to tell you how many subscribers clicked through to my Pinterest case study, I'd have to find every newsletter I've ever linked to it in and pull the stats for each link. And that would only tell me the number of click-throughs.
Because I was never able to get Aweber's Sale Tracking feature to work properly before I launched the study, I can't pull sales numbers – as tracked by Aweber – either. That's been on my list as a to-do item for a while, so I'll probably work on getting that situated over the next month, so I get some insight into the conversion rates for pop-up subscribers vs. subscribers via other methods. If and when I'm able to do that, I'll update this post with that data.
Another challenge in comparing conversion data is that I can't track most of my “conversions” from that granular of a level. My Pinterest case study is a clear-cut conversion that I can track via Aweber's sales tracking. However, comparing clicks and conversions for affiliate links, as well as AdSense – segmented by and matched up to my granular Aweber subscriber data – is not something I can do.
The bottom line
It would seem – for Sound Bites – there has been no significant difference between subscribers that subscribe via my pop–up vs. other subscription methods on the site. They exhibit almost identical behavior and engagement metrics. Considering that pop-ups have unarguably increased my sign ups without any loss of subscriber quality, I'll be continuing to use them on any site where building an email list is a primary goal.
As always, I recommend you run your own tests. These are the results for one site, in one niche. But this post should give you some insight into what metrics you can look at to make your own conclusions about what works for your site.
Do you have any data to share?
Have you taken the time to compare the quality of newsletter subscribers via various methods? If so, I'd love to hear what you've found in the comments below.
17 Comments
Please note – I use affiliate links on this site. This means I might earn a commission if you click on a link and sign up for something.
I love it when you said “Despite ‘most people’ claiming they hate them, pop-ups do work”, Rae – so true!
I wonder if anyone did a study on what kind of pop-ups work best, i.e. exit, after certain time on the page, etc… I am off to find one.
Thanks for publishing your findings; adding a pop-up to Traffic Generation Café now goes on my to-do list. ;)
Glad you found it useful! I compared the use of exit intent vs. time delayed pop-ups etc. in my OptinMonster review here: https://sugarrae.com/blogging/do-popups-work/ – hope it helps!
I discourage clients from using the Modal Method (I’ll write them in only if that’s what I’m paid to do, but I usually try to remove that option from the table as early into the build as possible) — Anything that flies out on to the page whilst the visitorship is doing their thing is a big no-no — The back button is our friend.
As I said – that’s what everyone tells me. But, the subscriber rates (and the lack of effect on bounce rates) say something else. And I’ve tested popups on multiple sites and over a period of years now. I’m not saying people need to use them – everyone should test how it works for their own readership / site. I think the “smartness” and strategy behind their implementation factors in to things as well (I don’t mind regular popups, but I hate the ones that hide the close button in almost blended text – and I’d venture a guess that those kind of popups do affect bounce rates when people can’t figure out how to close them).
I’d love to see hard metrics that agree with the everyone hates popups theory. I’ve heard all the detractions. I hate them. My mom hates them. They’ll damage your brand (never did manage to figure out how people came to “prove” that one). They’re less engaged. Etc. But I rarely see the data that resulted in those claims.
The few times I have seen people say they’re showing negative information on it, it’s only shown information from the popup, but not side-by-side with non popup data for the same metrics. i.e. someone might say “20% of popup subscribers never confirm. Popups suck.” with zero comparative data from non popup related subscriptions to back that up. Which, if I had the data to back comparatively prove that point, I’d sure include it (as I did above).
Also, I’d be remiss to point out that you didn’t hit the back button yourself, despite the disdain for popups, but instead read and commented. [I had to! LOL]
That said, if anyone has any information on popup usage showcasing negative results backed by data, I’d love to see them posted here. I’ve got no “dog in this race” so to speak. I can only report back on my own experiences. I’m using popups because my data says they work. I always love to see when there’s differing results, because that means I get to test more stuff. :)
Also, since tone is often lost on the Internet, my comments are meant to be an amiable rebuttal and not to dismiss. :)
Wow. Not even close to what I would have guessed. Thank you Rae for taking the time to share.
I know you’re a long time reader Rob, so, you know I always advise the following, but, to state it anyway: This is my experience. Your milage may vary. Everyone should always test things for their own site, in their own niche. :)
I’d be interested to see data from an e-commerce shop on this. My offering is “free information” – would be curious to see if paid products see similar results.
However, I’d want to see two segments compared on that – customers and non customers. So I’d want to see
Segment 1: customers who subscribed via popup vs. other methods
Segment 2: non customers who subscribed via popup vs. other methods
That would be interesting – would love to see a third comparison of the comparative numbers of subscribers who ticked the subscribe box on the order form compared to both popups and other methods as well.
Really enjoyed this research on a topic that I’ve long been curious about. I would be really interested in the click data and lifetime value.
Here’s a thought…what if you proposed to Aweber that if they get you the data (they’d probably have to do some manual SQL queries) that you’d publish it as a guest post on their blog. Just a thought.
I also wonder how pop-ups are performing over time. Anecdotally I have reached full saturation and just look for the way to make them go away. They could be offering me free Teslas and I wouldn’t even notice at this point.
Will look into it. :)
Great data, I was always suspecting popup to have bigger unsubscribe rate.
Even though I am on your newsletter list I saw this post on Facebook first so when your newsletter came to my inbox I had already read the post so the email got you an unopen from me. That I am sure is another set of data we never think about. In the end it doesn’t matter though because the end result is the same, I read the post.
Vinny – not sure I understood part of your comment – “I saw this post on Facebook first so when your newsletter came to my inbox I had already read the post so the email got you an unopen from me” – I think I’m missing something there. This post is a repurpose of something I shared in the newsletter, but with additional commentary, etc. But, my newsletter never shows the topics in the subject line, so you’d have had to open the newsletter to see that there was information on this topic in there. Unless you meant I didn’t receive a click from you in the newsletter – but, that wouldn’t affect the numbers in this case because there was no link to this post in that email? :)
I clicked through to your site because of the title .. Notice I didn’t subscribe [LOL] — *couldn’t resist*
We’ve already had this sort of discussion — If memory serves, back in 2003-04 there was a big debate over whether or not Browsers should be equipped with Pop-Up Blockers. Publishers were literally running off the rails with these things at the time and folks over at Microsoft, Netscape and the fledgling Mozilla started to take a long hard look at whatever options were available to them at the time.
Sure — There was the usual back and forth going on about it with each camp, for, and against, making claims to favor their respective sides.
Big Search also got involved. Jeeves, Teoma, Alta Vista, and even the fledgling Google got involved to various degrees on the matter.
It’s sort of ironic how it was Search that fired the first salvo in the Pop-Up Blocker wars – Google was breezing right along at over 200 million searches a day and it wanted to curb the pRon, spam, and interestingly enough, Pop-Ups.
The eventual outcome was that at the end of the day, Pop-Ups were bad.
In 2004-05 we started to see browser writes that included the Pop-Up Blocker option (there were already a few stand-alone plugins available at the time as well) — In 2006-07 we saw the Pop-Up Blocker installed and turned on by default in Internet Explorer, Firefox, Opera, SeaMonkey, and Safari, to name just a few.
The fact that the Pop-Up Blocker is installed and turned on by default in todays major browsers should be some kind of indication to publishers, as well as developers, that Pop-Ups are bad.
It took a few years before there were work-arounds to the Pop-Up Blocker — Enter the Modal window —
For those of you who may not know what a Modal window is:
In user interface design, a modal window is a graphical control element subordinate to an application’s main window which creates a mode where the main window can’t be used. The modal window is a child window that requires users to interact with it before it can return to operating the parent application, thus preventing the workflow on the application main window. Modal windows are often called heavy windows or modal dialogs because the window is often used to display a dialog box. ~ Wikipedia
As far as testing? .. Why not visit a bit of history about the Pop-Up itself — Studies were done on both sides with each side claiming victory — In the end however, it was the end user that voted with their feet, and both developers and publishers had no choice but to capitulate …
Pop-Ups are about as famous and about as well liked as FrameSets and Flash elements are, or as anything else is that interrupts the flow of a page — Most OnLoad, MouseOver, and other such things are funny and cute, but rarely are ever taken very seriously short of being an annoyance and shorting ones own conversions.
The disconnect is seemingly ever increasing again these days between developers and publishers — Case in point? – The modern equivalent to the Pop-Up discussions from the days of old is the AdBlocker of today — Soon, AdBlocker will be baked into the browser, turned on by default, just like the Pop-Up Blocker before it — The end user is once again voting with his feet — and in the not-so-distant future, we may also see C++ Browser writes that blocks out Modal windows as well.
SEO experts, Publishers and other Web Developers don’t pay my bills — The end user does – And if the end user suggests that something be bad, and the research according to that bad thing was already done a decade ago by companies with pockets much deeper than mine? — well then far be it from me to argue — We can do research and studies all day long on our own, or we can go to the research that big data did clear back in 2003 to draw our inevitable conclusions.
Oh, and by the way — Nice Site — and I mean that. I might suggest that if you have to use the Modal window (Pop-Up) – See if you can get your WebDev person to configure it to an “exit intent” deployment.
Have a Great Day — Never Give Up – Never Surrender ;)
The plugin I use, OptinMonster, already has exit-intent technology built in. I just wasn’t thrilled with the results I got when I tested using it in place of the timed popup. :)
“SEO experts, Publishers and other Web Developers don’t pay my bills — The end user does”
Same here. And according to my bounce rate, mine don’t seem to mind. :)
I had the theory that popup subscribers were somehow worth “less” than “regular” subscribers in terms of email opens, etc., so I ran a similar report and found similar results. In my case, the popup subscribers had a 3-4 ppt higher open rate than everyone else, which definitely surprised me.
Yeah – I think once you hear twenty people say “popups suck” with authority, despite never seeing any real data supporting it from sometime in the last decade, you just assume that it must be true. That’s why I always say test everything you hear.
Also, there is a big difference between the ad popup of 10 years ago that would launch ten popup ads when you left a site and the newsletter subscribe popup of today. I’m not sure why we keep seeing the old argument against *those* kinds of popups being offered as valid rebuttal in the newsletter subscribe popup debate of today. :)
Thank you so much! This article has been really helpful!
You’re welcome!