A study of 27 small business accounts
Professionals who’ve spent any time in paid search – at an agency or in-house – can tell you how much of a big deal Quality Score (QS) is to us in the industry.
We base our strategies on it.
We lie awake at night wondering how to improve it.
We use it as a justification for nearly every change we make.
Recently, Larry Kim at Wordstream took another look at our favorite obsession in his post “Revisiting the Economics of Quality Score: Why QS Is Up to 200% More Valuable in 2013.”
He isn’t the only one. If you do a Google search for any PPC firm + “quality score,” you’ll find veritable ass-loads of blog posts on it:
But nearly every study done on QS measures its impact on just one metric: Cost per Click (CPC).
It’s an important stat, don’t get me wrong – but looking at only CPC doesn’t matter much to our clients at the end of the day.
Why? For one, cheap traffic isn’t always good traffic. For a lot of businesses, the terms worth getting clicks on are the ones with the highest competition and the most cost-ineffective CPCs.
My point is: Our bosses and our clients don’t care about CPCs in and of themselves. They want more leads. They want cheaper leads. They want more-cheaper leads!
So, with that in mind, I wanted to look at QS under two different lenses: Conversion Rate (CR) and Cost per Acquisition (CPA).
The QS Study
We anonymously looked at 27 accounts with the following parameters:
- Had to be small businesses – after all, they stand to gain (and lose) the most from paid search
- Had to have conversion tracking enabled
- Keywords in the account had to have at least 1 impression
- Keywords with QS below 2 were ignored as there wasn’t enough volume to be statistically significant
That gave us a keyword base of ~40,000 in the study and an impression base of ~26,000,000.
We analyzed performance on that group over calendar year 2012.
The Results by QS
The first question we wanted to answer with this study:
Did getting a higher QS lead to better CR?
The answer might surprise you:
Google Shopping Campaign Screenshot
*Note – Product categories and labels have been altered for privacy reasons.
No, not really.
It makes sense that there’s a huge jump going from 2 to 3 QS. That’s basically the difference between your ads showing up often and not showing up at all.
But improving from 3 all the way up to 9 has literally no Conversion Rate benefit!
You only see massive benefit in Conversion Rate moving up from 9 to 10.
The next question was:
Did getting a higher QS lead to better CPA?
This answer was less surprising:
Yep. It’s a massive deal.
QS can cut your conversion costs by over 90% improving from 2 to 10.
But even going from 6 to 10 can improve your conversion costs by over 50%!
Here’s the data table, if you’d like to just stew in the numbers for a minute here:
QS Study Takeaways
The question you really need to be asking – QS aside – is:
What are your client’s goals?
Do they want more conversions?
Being QS obsessed, or doing bid adjustments based on QS might not be the way to go if the client just wants a better conversion volume.
In fact, this study saw a greater CR at QS4 than any other besides QS10.
Do they want cheaper conversions?
This is where QS has to be your absolute mission. The benefits to CPA by even incremental QS improvements were incredible.
Improving your QS by 1 meant an average CPA reduction of 22%!
Did you segment by brand v non-brand kws? Conversion rate / CPA for branded terms, which typically have much higher quality scores, is going to be significantly different.
That’s a fair point, Jason.
I didn’t for this study.
Mainly because these were small businesses that didn’t have huge search audiences for their brand terms.
The balance of Non-Brand to Brand traffic was probably 95%/5% – not enough to skew the numbers significantly, I felt.
When you say, “But even going from 6 to 10 can improve your conversion costs by over 50%!” are you suggesting that you were able to improve keyword quality scores from 6 to 10 for that result, or are you simply comparing a keyword with a QS of 6 to a keyword with a QS of 10? Thanks!
We looked at a bunch of keywords at each of these scores on aggregate. Charting their progression over time would be an even more fascinating study, but I’m not sure I’ve seen anybody do that yet.
Thanks, and agreed. My follow question would have been, damn, what were your methods to achieve those quality score increases !
Small sample but this is consistent with what I have found. That quality score reduces cost per click, cost per conversion, and increases conversion rate. For this reason I believe quality score deniers are just bad PPC marketers that aren’t acting in the best interests of their customers.
Amen.
Strong words Larry. And I agree with them. I am curious what the average monthly spend is for these 27 accounts and what the average number of monthly conversions is for these accounts. Great study and proves my shared belief with Larry that QS is critical.
Eventually, I’ll expand this study out to our mid size and enterprise client base, but PPC can make or break small businesses online – so I wanted to start there.
I appreciate the feedback, Larry.
As Jason pointed out above, I think it is important that branded keywords are removed from this study, as they will heavily skew conversion rates and qualities scores.
That being said, the data around the small conversion rate is quite interesting. I would have expected it to be a little bit more of a variance.
As for the CPA data, I am not sure if this is more correlation or causation. While your data above points to your theory being correct, I can dive into an account that bids on competitors brand names that have QS in the 2-4 range that convert at better CPA’s then some of our unbranded keywords with QS’s in the 6-8 range.
Don’t get me wrong, I understand QS is important, but it is not the end all be all metric. Your customer’s bottom line and profit is. Another level of data such as revenue generated from sales/leads would really show some pretty cool insight.
Very cool study – thanks for pulling this together.
Again though, the Non-Brand balance was about 95%. These are small businesses with virtually no Brand awareness.
Totally feel you on correlation vs. causation. That’s the rub with any PPC study, as there are so many moving parts.
Good study Michael, thanks for sharing this.
You say ” but looking at only CPC doesn’t matter much to our clients at the end of the day”.
But if the conversion rate is not really effected, than the CPC must be the only other variable resulting in a lower CPA. Since CPA = CPC / CR% .
So this study even proves that a lower CPC focus (ceteris paribus) is not really a bad thing in QS optimizing strategies, isn’t it?
That was my point in regard to CPA.
I was talking more in the context of conversion volume that QS isn’t as important.
If your client is willing to pay a premium for more leads, QS4 or QS9 will be of little consequence to them.
I’m glad I found your post, I was only just today going through my own account data looking at the affect QS has on CR.
In my small sample with brand keywords removed, I found higher QS led to higher CR and lower CPA. This has led me to pause many keywords below QS7 as the CPAs were too far off of the target.
I’ve found this a good method when starting a campaign to get a good footing, once the campaign is profitable I can then open it up to QS6 keywords and so on.
Quality score is just another indicator. There are times when lower quality scores need to be tolerated to get the result for a client. Good example is “removalists” $8-$10 cpc. My client is not the cheapest, actually &10 ph more than competitors. I use price in ad to prequalify, which does effect CTR and quality score. It works even though quality scores are 5-6 we are able to get high conversion rate and more importantly high conversion from lead to sale. Understanding your leverage points is the key and one size doesn’t fit all
That’s a fantastic point, Jayson. Many, many B2B clients use ad text to pre-qualify leads, which has a direct negative impact on QS, even though it’s positive for their business.
Absolutely need to understand the client’s goal first before you go QS crazy. It’s paramount.
How can you say that QS has an impact on CR? I get the CPA piece – as QS has an effect on your cost. But how does going from 9 to 10 change the conversion rate? Can’t see why this is possible – conversion rate changes are more likely to be due to ad copy / landing page changes rather than QS. Would like a bit more clarity on this 🙂
CRO changes absolutely have an affect on QS, whether directly or indirectly.
Some examples:
Direct – Landing page quality affects QS.
Indirect – Ad copy affects CTR which affects QS.
Just food for thought. Thanks for the comment, Dan!