302-pepper-spray
The pain of a 302 response
Over the last three months, a bunch of folks have used my server response code tester to check whether their web sites correctly respond to broken links, correct links, etc. Per my first post about the tool, I’ve been storing the results—don’t worry, no personal data, just web site, response codes, and a grade—for a survey.
The short version: We all get a C-. I don’t know about you, but when I got a C- on a report card, it was 1 week of forced labor, doing homework inside while my friends frolicked happily in suburban paradise.
Server response survey results: Now, with more pepper spray!
Response code survey results: Who got it right
Here are the details: What I found, how I found it, and what it all means:
Test method
The response tester looks at four things:
- How your server responds when a browser visits your domain name at www.yoursite.com. It should deliver a 200 or, in some cases, a 301 response. Anything else is likely a problem.
- The tool checked the response for www.yoursite.com/sfghawerhascd.aspx. That’s a .NET page. Unless you’ve actually built a page with the address ‘sfghawerhascd’, your server should respond with a 404 code.
- It checked the response for sfghawerhascd.php. That’s a PHP page.
- And, it checked the response for sfghawerhascd.htm.
Folks tested over 2000 sites using the tool. I removed duplicates and then ran the numbers. The results are pretty interesting.
We’re all doomed
DOOMED, I tell you. Apparently, only 69% of all ‘professional’ webmasters/developers can correctly configure a web server.
The internet, as a whole, gets a D+. That means that internet marketers, as a whole, get a D+. Don’t give me that oh-this-is-technical-stuff-for-developers crap. If you can’t figure out how to correctly configure a server, you should at least be able to harass someone else until they do it.
Of 2000 servers tested, only 69% got an ‘A+’, which I defined as:
- A server that delivers a ‘200’ response code for the home page…
- A ‘404’ code for a broken link pointing at an ASPX .NET page…
- A ‘404’ code for a broken link pointing at a PHP page…
- …and a ‘404’ code for a broken link pointing at an HTML page on their site.
A D+. Next time some code nerd tries to get all high-and-mighty and cynical on you, run the response tester. If they don’t get an A+, slap them, really hard.
It is ridiculous that 30% of servers get this wrong. Asinine. Stupid. Silly. Crazy. Insane.
Here’s where the relative level of difficulty lies:
Impossible: Understanding US tax law
Really hard: Winning the Tour de France
Hard: Riding a bicycle 100 miles in a day
Medium: Riding a bicycle to work without being squashed
Easy: Riding a bicycle in a wobbly circle in your driveway
Really easy: Configuring your server to deliver the right response code
So yeah. If we can’t get this right at least 90% of the time, we’re doomed.
.NET gets an F-
.NET servers were correctly configured only 22% of the time.
Twenty. Two. Percent.
My. God. I’m a Mets fan, and 22/100 is still shocking.
548 sites tested were .NET-driven. Each site had 4 chances to get it right, for a total of 2192 chances. Total, they scored 503 points.
By the way, there’s no explanation for .NET sites’ utter awfulness. Except for the fact that .NET seems to attract awful developers like bright lights attract insects.
Mountain, not molehill
This is a mountain, not a molehill. If your server’s delivering the wrong response codes, you’re clobbering your SEO potential, and may be hurting site performance as well.
Here’s an example: When a web browser or search bot visits your home page, the best response is a 200 code. There are a few exceptions, but not many.
200 sites tested delivered a 302 response code for www.yoursite.com. I won’t explain why that’s bad. I’ve already explained it here.
When it comes to SEO, a 302 redirect on your home page is hari-kari. You’re telling search engines “Nothing to see here, move along, and don’t bother with any authority, either.” It’s a cosmic flush of the SEO toilet.
Lots of others used a 301 redirect from www.yoursite.com to /index.html or similar. That’s slightly better. Slightly.
Only 73% of servers tested delivered a 200 response.
If only 73% of drive-throughs delivered the right fast food, we’d have riots in the streets. All you have to do is ask someone to change a couple of server settings. You probably won’t even get pepper sprayed.
I’m moving to Mars
When I went into this project, I figured I’d find, I dunno, 10-20% of servers slightly messed up. Instead, I found 1/3 of the web well and truly bolluxed. I’m not sure if I should shout with joy because this means job security, or scream with despair because 30% of webmasters aren’t smart enough to breathe if they get dizzy.
On the Ian Scale, I’m moving to Mars, or opening a bicycle shop:
Server response survey results: Now, with more pepper spray!
Mars or a bike shop: It's my choice
Mars is cold this time of year, and bicycle shops mean (shudder) working in retail again.
Please. Fix your servers.
That’s pathetic. But thank you for making such a useful tool – I’ve used it many times to prove to a server admin that their server was indeed all messed up.
“.NET servers were correctly configured only 22% of the time.”
(because corporations use .NET)
*says no more*
Whenever I see a weird home-page canonicalization problem, I can lay odds on it being .NET server even before I see the file extensions. It’s almost like Microsoft designed it to harm SEO on purpose.
How would you suggest handling an all https website? I throw 301s from http to https for all addresses, so the 404 is delayed until after that 301.
That should be fine. At least, it’s the best option I’ve come up with.
In defense of developers, most (certainly those in corporations) have absolutely no say in web server configuration. That falls on the sys admin.
And the reason that .NET sites are worse than PHP sites is that IIS is easier to screw up than Apache and it’s harder to detect those screw ups with IIS than it is with Apache.
My feeling is that one of the reasons for this mess is the origin of ASP.NET. It has never been meant to be a framework to produce perfectly valid HTML. It has been created to develop web applications in a way that is not too different from developing client server applications. That whole ViewState business is a mess which is meant to be pushing some kind of stateful protocol on top of the stateless HTTP.
I cannot count how many times I’ve seen requests to non-existent URLs being 301’ed to error pages which display in big letters “404 – Page not found” and return a response code of 200. And usually those “web developers” then reluctantly change the code of their error page so that it returns a 404 when asked to, still doubting the need to do so.
ASP.NET does not care much about web standards, it’s just build so that it looks ok for the user – in his IE6…
Does the tool check to see if alleged 404 pages return a 301 or 302 like many out of date SEOs used to recommend?
Hi David,
Yes, it does
Sweet. Our whole range of websites get an A+ 🙂
Don´t pack your bags just yet, Ian.
I got an A+, I must be the class geek!
Nice tool 🙂 A+ for my sites.
it doesn’t seem to work in IE8 tho.
I’m no techie. In fact I’m not totally following everything in the article. Isn’t it my hosting company that is responsible for my server? I have two websites hosted with GoDaddy. One gets an A+ and the other gets a D. What can I do about this?
Hi Glen,
Send ’em the info! Most hosting providers have great tech support, and they’ll either fix it, or tell you how to fix it.
Ian
Correction. .NET servers don’t respond incorrectly, the rest of the internet is wrong. Microsoft products don’t follow “standards” because Microsoft knows best. Just ask any who has ever tried to design a website and has had to resort to hacks for the other inferior browsers because IE handles CSS and JavaScript the right way…
Of course, Gary, you’re totally right 🙂
Awesome comment.
Wow, I read the whole way down through your post and really you need to calm down. Alot the time seo people like yourself read a book or two and think you know everything about technical stuff.
Not saying the technical people who are in charge of their servers are perfect, but unless both seo people and technical people work as a team. Then these are the kind of numbers you will get.
We servers are not that difficult to configure, provided you know what you are doing. That is the problem, there is a lot webmasters who don’t have a clue on what they are doing…they read about from a book or online. And think they know it.
I prefer IIS as my we server, and always consider seo when setting up a new site for the first time. However, if it is an existing site being replaced, then it is the first flag I raise. Which is where do you want the old links on the site to go to in the new one. You would be amazed at how many times people have looked at me and told me. It doesn’t,t matter.
And that is when they complain that people are getting 404s and their search ranking has gone down. Everyone has to work as a team, and get off their high horse about stats like these.
I ran it on two of our sites and got A+ (yay!). But when I ran it on a third, it generated an error page at this URL: http://responsetester.appspot.com/whatifinate. Here’s a screenshot:
Hi Miriam,
See my reply to Ostheimer Webdesign – sorry about that.
For my website I get this using the tool:
Traceback (most recent call last):
…
DeadlineExceededError: ApplicationError: 5
I’m still working on the error handling, sorry. That means the tester ‘timed out’ – it took too long for your site to respond with a code, so Google App Engine gave up. Try running it again to see if it keeps happening. If it does, you may have latency problem on your server.
Nice tool, thank you for posting it here!
I just used the Server Response Code Checker to test one of my servers hosted at Bluehost and got an A rating.
Then I used the same tool to test one of my own servers that I built and host on my own and got an A+ rating.
Your tool has just fluffed up my ego a little bit. 🙂
Glad I can help your self esteem 🙂
Your article got featured in the seomoz newsletter. Andre said almost exactly what I was about to write, except that our custom built servers are located at Dimenoc, all got a A+ Rating! What a nice start into the week 🙂
Woo hoo! Didn’t see the feature. I’ll go look.
Is there not a valid reason to respond to all requests to your domain with an actual page? If a user has misspelled the url, or followed an out of date link, surely its better for them to see an error page, from which they can navigate to the appropriate resource? Every visitor is a potential conversion, regardless of how they arrived at your domain. What are your thoughts?
Definitely, but a quality 404 page counts as a response. A good 404 page should make it clear to the user that something went wrong, but also provide clear call to action, so they stick around.