Analytics Apples And Oranges: Switching Web Analytics Tools Without Getting Fired

I did WebTrends a disservice here by failing to point out that my client’s creaky WebTrends installation was a very old, purely log-file-based one. Newer versions of WebTrends are every bit as versatile and accurate their competitors.

If you switch from one web analytics package to another, be ready to make some adjustments in your metrics goals.

Web Analytics Q & A

How many people here used WebTrends or something similar up until a few years ago?

How many switched to, say, Google Analytics, Omniture or Urchin 6?

OK, last question: How many of you nearly got fired when your visitors, pageviews or other metrics inexplicably plunged?

Chances are you aren’t suffering from bad statistics. You’re a victim of metrics madness: The lack of standards in site traffic measurement.

The Horror of Metric Madness

Late last year we launched a shiny new web site for a client. It’s beautiful, if I may say so.
As part of that launch, we switched them from a creaky, old, log-file-only WebTrends installation to Google Analytics (note my comment at the beginning – WebTrends is great. Just don’t use a 5+ year-old version).
Four weeks ago I got a wake-up call. Pageviews and unique visits to the site had dropped 10% over the previous year.
December 2006: 300,000 pageviews
December 2007: 270,000 pageviews
(numbers changed to protect the innocent, percentages the same)
For this client that’s pretty grim: They’d seen steady growth (thanks in part to SEO) before 2008.

Those Missing Pageviews

So, where did those pageviews go? And come to mention it, why didn’t they go up, instead of down? Have I lost my touch?
Nope.
We’d switched the client from WebTrends, a tool that measured traffic using log files and nothing else, to Google Analytics, which measures traffic using a fancy javascript.

The Difference Between Log and Javascript Tracking

Warning: Geeky stuff ahead. If your eyes roll back in your head at words like ‘log file analysis’, skip ahead to ‘Analytics Package Conversion Factors’.

The WebTrends installation my client had used the server logs to count visits and pageviews. It counted any page load by any visitor as a pageview. Even if the visitor started to view the page and then clicked away before a full pageload.
Google Analytics, on the other hand, waits until the entire page loads. The javascript that Google Analytics uses to count a pageview loads at the bottom of the page. If the whole page doesn’t load then Google won’t count it. Visiting search engine spiders and other non-javascript browsers don’t execute the javascript at all, so those pageviews never get counted.
Third-party tools like Compete.com further reduce pageviews and visits because they’re not measuring 100% of traffic to your site. They get a random sampling based on either traffic through your ISP or toolbar installations (like Alexa.com).
All of this can cause a perceived but artificial drop of as much as 50% in both pageviews and visitors. It’s often less, but it depends on the original configuration. Here are some rough percentages based on my experience:

Web Analytics Package Conversion Factors

These numbers are from sites I’ve worked on, not the entire internet. While I’ve been around the block more times than I care to count, take ’em with a grain of salt.

From Urchin using UTM.js to Google Analytics: 20% drop
From Log file-only WebTrends or Urchin to Google Analytics: 30-50% drop
From Webalyzer to Google Analytics: 20-30% drop
From Google Analytics to Compete.com: 20-60% drop
3rd-party sites like Compete.com are tougher: A busier, more popular web site may see only a small drop. A smaller, less popular one may see a very large drop. A smaller initial audience leads to a smaller sample and a larger drop.
Generally, you’ll see an apparent drop in pageviews and visits whenever you switch from a logfile-based analytics tool to a javascript ‘bug’ based tool like Google Analytics. And you’ll see another drop when you compare javascript-based analytics to a 3rd party toolbar or internet service provider-based tool like Compete.com.

Two Tests To Save Your Job

So, your boss or client is really pissed. It looks like you’ve cost them a big chunk of traffic. Your life, or at least your career, is about to come to a messy end. There’s plastic spread on the boss’s floor.
Here’s how you save yourself:

Use Compete.com

Go to Compete.com. Look at the trend over the time period in question.
For example, let’s say I moved my blog from Urchin to Google Analytics January 1, 2008. My own data shows a drop in visits:

These numbers are made up, by the way.

I take a deep breath, go to Compete and look at the same period:

Ah HAH! My pageviews didn’t drop. I’m just seeing the effects of a switchover from log- to javascript-based tracking.

Run a Logfile Tool On Current Data

If you have access to the log files on your web site, why not point a log file analysis tool like Urchin or WebTrends at the logs? Then you can compare your visitors and pageviews across multiple analytics packages, figure out the difference and adjust accordingly.
Chances are your web hosting provider already has some form of basic log file analysis running on your server. It might be pretty crude, but you only need the most basic stats: Visits and page views.

Don’t Panic

Different analytics packages measure different ways. There is no central standard for web traffic analysis.
So, if you move from one web analytics tool to another, and your numbers plunge, do a little double-checking before you panic.