Keyword research used to be so easy. You picked terms that the client wanted to rank high for, stuffed them onto the page, shake, bake and voila, the site is ranking in the top 3 search results for query.

Bad SEOs stuffed keywords in all of the wrong places, all over the page, and in same font color as the page background so that only the search spiders saw them. Good SEOs stuffed them into anchor text, heading and subheadings, and <title> tags.

And then things began to change: first in a good way, then in a not-so-good-way, and now maybe in a good way again.

A Brief History of Search Engines

Keyword research was easy because search engines were so dumb – I mean “fleece your little brother for his paper route money” dumb.

All they had going for them was the flimsy Term Frequency/Inverse Document Frequency formula that rewarded documents that had the most instances of the query terms in body text, along with some lightweight suppression clause so that long boring documents did not always get the top spots.  Yay early information retrieval.

The good days of gaming search engines came to a screeching halt with Google’s buzzkill PageRank algorithm. PageRank was based on the academic model that stipulated papers cited by other papers had to be better than those not so cited. Applied to the decidedly non-academic public Web, pages that had a lot of links pointing to them had to be more relevant than the others, right?

Peace and relevant search results lasted until the SEO community was able to figure out how to game this system with begging, borrowing and buying links. Google was shocked! Evidently, this did not happen in the hallowed halls of academia, at least not in the Stanford Graduate Computer Science program of the 1990s.

Waaaaaahlll Pilgrim, Google was not going to take that sitting down. (I have it on good authority that they take nothing sitting down because those stand-up treadmill desks are standard issue at the Googleplex.) They fire off the Hilltop Algorithm. Hilltop was one of the first algorithms to introduce the concept of machine-mediated “authority” to combat the human manipulation of results for commercial gain.

With Hilltop:

• Pages are ranked according to the number of non-affiliated “experts” that point to it, i.e. not in the same site or directory
• Authorities have lots of unaffiliated expert document on the same subject pointing to them
• Affiliation is transitive [if A=B and B=C then A=C]

The beauty of Hilltop is that unlike PageRank, it is query-specific and reinforces the relationship between the page and the user’s query. You don’t have to be big or have a thousand links from auto parts sites to be an “authority” and float to the top. And, to the rejoicing of searchers, it was soon adopted by the other search engines across the land.

Giddy with the success of contextual mapping, the search engines followed up with Topic-Sensitive PageRank.  This involved the geekiest of information retrieval methods, use of predictive analytics, and vector space modeling on a subset of the Web to analyze the context of query phrase term use in a document, in the history of all queries, and in the history of the user who submits the query.

As if Christmas in July was not enough for searchers, the search engines also laid down ontology (an organized schema of subject categories) supposedly derived from the Open Source Directory.  I don’t know about you but that looks a lot like the Semantic Web to me.

Ironically, as search engines got smarter, searchers got dumber. Most of them started to construct poor queries (56%), select irrelevant results (55%), and become disoriented and overwhelmed by the amount of information in search results (38%). Hmmm…maybe it is time to start taking a user-centered approach to optimizing websites for users?

User-Centered Keyword Research

User-centered keyword research lives up to its name by starting with what prospective customers would likely use to find the site. And the best place to find that information is your client.  Familiarize yourself with the client’s product space and vocabulary, ask questions, and look at their competitors.

Then, turn to Google Analytics to find out what is sending traffic and how it is performing. I look at what page they land on, whether they engage or bounce, and if they convert. If there is one tail, long or short, in SEO that is supported by data, “longer query = more likely to convert” is it.

Finally, swing by Google Webmaster tools and see how the search engine currently views site relevance by studying queries, impressions, AVERAGE (important distinction there) position, and click-through rate.

Next, compare actual site behavior with general search behavior using any one or all of my favorite tools:

Google Trends is not nearly a big enough return for the egregious and persistent invasion of our privacy, but it comes pretty close. Google Trends is a view into the long, dark, deep Google data mine of search behavior with the capacity to filter by geography or time.

The true delight lies in seeing Top and Rising search queries to the term phrases in comparison. It provides actual user search behavior. In the comparison below, the phrase “user experience” is more popular than “information architecture.” Note that a significant portion of the search around the general phrase is job-related based on the Top Searches information.

Google Trends

Yahoo! Clues offers many of the same data points as Google Trends with demographic information (age, gender) thrown in for good measure. The data is extracted from Yahoo! Search and is aggregated and anonymized. A Yahoo! Clues-specific feature is the Search Flow data that reveals what the user searched for before the term phase comparison and what they searched for after.

Yahoo Clues

We’ve all experienced the mostly annoying yet occasionally helpful search suggest, the list of query suggestions that appears as you start typing, and changes to meet the changes in your query.

Ubersuggest provides an easy to navigate, much less annoying, more useful aggregation of search suggestions from Google and other “suggest services.” In looking at the Ubersuggest results for query keyword research, I’d say that most folks want someone or something else to do the work for them.

Ubersuggest

As you can see from Ubersuggest, if you are looking for a keyword research tool, you are not alone in your quest. Which you choose, however, will be up to you. Some perennial sites in the top search results for “keyword tool” are:

• Google Adwords tool
• Bing Keyword Research tool (requires login as part of Bing Webmasters account)
• Wordstream
• SEM Rush
• Wordtracker
• Keyword Spy

Magical Thinking with Psychographics

At a search conference in July 2012, Marty Weintraub from aimClear delivered a groundbreaking presentation on using Facebook psychographics to develop a new type of user persona that can assist with remarketing.

On the aimClear blog, psychographics are defined as: “…a means of identifying users by interests, occupations, roles in life, predilections, and other personal characteristics” This involves mining social outlets for personal preference data, e.g. a political reporting website that targets individuals who listen to Rachel Maddow, Stephen Colbert and Al Jareeza, like the Muppets, and work for a middle of the road or left-leaning online or print publication.

These preferences are often articulated with term phrases developed by users and potentially reveal what they would use when looking for the client product while facing a search box.

In his book “How Buildings Learn”, architect Stewart Brand recommends waiting a few weeks after a building is finished before putting in the walkways. His reasoning? The footprints in the grass will tell you where people are walking to get in and out. So it is with smarter keyword research. Before stuffing a bunch of term phrases on a page, start with what searchers are using to find your client’s product or service. Then keyword research will be as smart as the search engines, or even smarter.

What keyword research tool do you find most helpful?  Let us know in the comments below.

The post The Evolution of Advanced Keyword Research appeared first on Portent.

No, not Frodo. His will shredded like wet toilet paper. Gollum took the lava bath, destroyed the ring, and saved everyone from a 36″ Dark Lord. Does he get any credit? Nooooooo. Nine-fingered Frodo is the hero. The ladies all swoon at Legolas and Aragorn. But not poor Gollum.

Internet marketers, and SEOs in particular, are a lot like Gollum: We’re shunned. We don’t get much sunlight. Our diet is awful. And we never get the credit for business success.

It’s our fault. We’re good at building rankings and building traffic. But we’re terrible at demonstrating the value of our work. So we fling ourselves into the lava. Every single time.

The way to avoid the lava swan dive? Collect big data. Present simple analysis. Don’t deliver reams of data. Use those reams of data to generate insightful, at-a-glance analysis.

I’m going to walk through whole process, from data collection to analysis and presentation, in this post:

Our sample company

Nazgul Cyclery. Cycling through Mordor? We’ve got your rigs!

None of my clients wanted to be compared with ravening hordes of goblins and uruk, for some reason. So, I’m going to use a fake company: — Nazgul Cyclery — as an example throughout this post. Nazgul is the world’s largest bicycle shop chain. We do 10% of our business online, and the rest off. Competitors tend to, er, encounter problems, so we’ve grown fast.

We’re best known for our chainrings. There are 9 of them. Plus one forged in darkness… Never mind.

First, collect your tools

You’re going to want some tools to do all the stuff in this post. My suggestions:

• At least one really good authority measurement toolset. SEOMOZ (affiliate link), MajesticSEO and/or ahrefs are all great.
• A site crawler. I like Screaming Frog SEO Spider. Xenu Link Sleuth is great, too.
• Google and/or Bing Webmaster Tools.
• Analytics with goal tracking.
• A rank tracking tool. Yes, we still have to look at the rankings. They go up and down. They vary. They get [not provided]. But face it: The higher ranked stuff gets more clicks. I use AuthorityLabs and Advanced Web Ranking, but SEOMOZ’s toolset can do it for you, too.
• A basic keyword research tool. For our purposes, Adwords’ keyword tool will work OK. You can use Wordtracker, Wordze, KeywordDiscovery or something else, if you prefer.

1: Grade yourself

First, you have to answer the question: How are we doing right now?

Create a report card for your web site. Collect and track stuff that influences rankings:

1. Number of duplicate pages on your site.
2. Number of 404 errors.
3. Number of 200 status pages.
4. Number of indexed pages.
5. Home page load time.
6. Inlinking unique domains.
7. Keyword diversity.
8. Pages with no title tag.
9. Pages with duplicate title tags.
10. ‘Thin’ pages.
11. Domain authority from Open Site Explorer, Majestic SEO or ahrefs.
12. Number of top 10 rankings.
13. Klout score, as a general social media indicator.

Stuff that doesn’t influence rankings, but you should still track:

1. Unique visits from organic search.
2. Non-branded unique visits from organic search. These are folks coming to your site searching on phrases that have nothing to do with your brand name or products. Someone finding my bicycle shop after a search for “fix my stupid flat tire” is a non-branded search.
3. Time on site from organic search.
4. Time on site from non-branded organic search.
5. Average pageviews from organic search.
6. Average pageviews from non-branded organic search.
7. Bounce rate from organic search.
8. You guessed it: Bounce rate from non-branded organic search.
9. Conversions from organic and non-branded organic search. This one is really important. You must have some form of conversion. There’s always one — folks who contact you, a purchase, an information request. Whatever it is, track it and assign a dollar value to it.
10. Dollar value of organic search. Also really important.

More on goals for a second: I only sell 10% of my stuff online. I’d love to do more, but my shops are important, too. So I’ll track a few things: Web sales, visits longer than 6 minutes (the site average), social media interactions and ‘find a store’ searches.

I use a spreadsheet to track it all. If you want to use my example sheet, click here. The end result doesn’t have to be pretty. This is just storage:

Store everything you think you’ll need later

Collect everything you ever think you’ll need. Ever. You can’t go collect it later. You want a big, happy data warehouse you can use later.

2: Build the opportunity map

I’ve talked a lot about SEO opportunity gap research. Here’s a quick summary:

1. Find phrases that generate traffic and conversions, right now.
2. If there aren’t any, find phrases that generate traffic, right now.
3. Check your rankings for those phrases.
4. If you rank between the #4 and #20 positions for a phrase, record it in a spreadsheet.
5. Map it: Record the page for which you best rank for that phrase.
6. Record the current traffic volume.
7. Record the traffic volume according to your keyword research tool.
8. Figure out how many more visits you’d get if you got to position 1, 2, 3 or anything else better than current. You can use Slingshot SEO’s current research for this.
9. Based on conversion rate, volume and potential ranking, calculate how many conversions you can gain with an upward jolt.

Now, do the same thing for top key phrases for which you have no ranking at all. When you map these phrases, you won’t know the current traffic volume or the mapped page. That’s OK — just record ’em anyway.

The result is an opportunity map, showing phrases, current rankings, current conversions/revenue and the potential lift for each. I used a little conditional formatting to highlight terms that may offer the most promise:

The opportunity map

Important: Yes, I’m using keywords to do this. But I’m using keywords to figure out my site’s strengths and weaknesses. I will not be telling clients “We are going to rank for blah blah.” Use keywords as a comparison tool, not a success metric. Otherwise, when keyword data goes bye-bye forever, you’ll be hosed.

3: Estimate difficulty

How hard is it going to be to make the gains you’ve mapped out? Grade that, too, using these factors:

1. Number of competing pages with the same phrase in their title tag.
2. Domain authority for the top 10 competitors.
3. Social media metrics for the top 10 competitors.
4. CPC for each phrase.
5. Inlinking unique domains for the top 10 competitors.

I tend to do this at-a-glance, by eyeballing some of my competition data for keywords:

Competition data in the opportunity map

Then I look at competition data by site, too:

Phrase data by site and SERP

If you want to get fancy, you can also track your competitors’ issues:

1. Duplicate pages.
2. 404 errors.
3. Thin pages.
4. 302 redirects.
5. Server response code handling.

That can help you determine where you may have a competitive advantage.

This can get tedious. Learning a little Python or another scripting language will save you a lot of time. At Portent I wrote a script that grabs a lot of this for me. It saves us about 1 hour per report.

I actually assign a 1-10 difficulty score based on what I see. This is a holistic, numeric grade I assign by just looking. I find it a lot easier to organize my work once I’ve done this.

Remember: Everything we’ve collected to this point should never see the light of day. It’s your data, for your use. Don’t lob it at the client.

4: Track SEO goals based on your grades

Start tracking goals based on what you’ve found. There’s no hard and fast process for this, but I usually:

1. Find the best possible conversion-generating opportunities from step 2.
2. Look at my site’s weaknesses based on step 1.
3. Look at my site’s strengths based on step 1.
4. Look at competition for each opportunity, based on step 3.

Then I sort and prioritize. Some ultra-competitive phrases may offer huge traffic and conversion gains. Those become long-term goals. Other less-competitive phrases may offer solid traffic and conversion gains. Those become my near-term goals.

5: Map actions to opportunities, based on your grades

What’s it going to take to move up for these goals and grades? You now have a lot of data. You can use it to make some pretty solid decisions. Look at each opportunity in your map. What do you need to do to pass your competitors?

You need to map specific actions to these opportunities, so that your team knows what to do, and the potential impact.

• Great link profile + thousands of duplicate pages = You’d better fix those duplicates. Provide specific recommendations for fixing each duplication problem. Track completed recommendations.
• Thin content + competitors with great content = Content strategy. Provide an editorial calendar and publication guidelines. Track launched content and response.
• Lousy authority + competitors with high authority = Social media and link acquisition. Provide curation guidelines and start a link campaign. Track links acquired and social media profile growth.

Again, look at how I’m using the keyword data: Not to measure success. Instead, I’m using it to drive decisions about next steps.

This is the first bit of information you’ll show the client. This is the ‘what’—what you’re going to have to do to grow.

6: Track it all

Every day, update your tracking sheet. If you’re short on time, do it every week. Or, find a way to automate it all.

My favorite tools for automation are Google App Script and Python. Use what you like.

Whatever you use, make sure you update your site and goal data on a regular basis. A real masochist will also update the data on all competitor sites. But I train using a video series called TheSufferfest and I still don’t do that, so it’s totally your call. If you have underlings to do your bidding, go for it.

7: Create the report card

All of the work you’ve done so far sets the stage for the report you’ll show your bosses. Right now, you probably have spreadsheets with a lot of columns and even more rows. You can’t send that to the Nazgul Cyclery VP of Marketing. I hear he’s a real hardass:

My clients are all cats, so I’m in really good shape. I won’t bury them in all the data because I like them.

You want the report card to link action to results. That’s the entire purpose of this exercise. So my dashboard would include:

• Sales.
• Change in SEO issues, such as duplicate pages. Improvement should mean good things, after all.

Here’s my result:

The dashboard – all that work, for this?!

That’s it. Keep it simple. If you need a more complex dash, go to step 8.

A ‘report card’ is just a style of dashboard. There are lots of great resources on creating good dashboards. Take a look at the examples on http://patternry.com/p=information-dashboard/ for starters.

8: Layer your dashboards

You may have multiple layers of bosses.

If so, create additional, more complex dashboards as needed. Track changes in individual social metrics, keyword diversity and other stuff. Allow your audience to drill down through the data.

9: Don’t report. Analyze

Don’t just hand over the report every week/month!!! Little misunderstandings can lead to big consequences.

Include good, useful analysis.

Good analysis:

• “We fixed all duplicate content issues and added 10 inlinking domains last month. We now have 300 more indexed pages, and our domain authority rose 2 points. The result? We’ve got 10% more non-branded traffic and sales from search are up 2%. Next action: Step up the content campaign and start our next Smeagol Sweepstakes.”
• “Our site had a serious duplication issue last month. That meant more indexed pages, but poorer rankings. As a result, sales fell 1%. Next action: Fix the duplication issues. Potentially dump guilty parties into Mt. Doom.”
• “The last blog post we did was featured on Overlord Weekly! We gained 25 inlinking domains. That hasn’t kicked up the rankings yet, but it will. Next action: Figure out why that particular post attracted so much attention and adjust our strategy. Also, let folks know by publicly thanking Overlord.”

Bad analysis:

• “Rankings are up. Sales are up.”
• “Traffic fell 10%, but we expect it to recover.”
• “We’re all doomed!!!! Dooooommmmmmmeeeeed I tell you!!!”

Last advice

Don’t be afraid to brag! Most SEO’s (including me) are so paranoid about over-promising we never take credit for anything. Get over it. Take credit when it’s due. One great way to do that is to give credit, like this:

“Thanks to the dev team, who put in a lot of extra time with me to figure out the duplication issues, we fixed all duplicate content issues and added 10 inlinking domains last month. We now have 300 more indexed pages, and our domain authority rose 2 points. The result? We’ve got 10% more non-branded traffic and sales from search are up 2%. Next action: Step up the content campaign and start our next Smeagol Sweepstakes.”

Also, put everything in context. Say the company is investing $50,000 a month in content strategy (I stifled a giggle right there). Your team produces 25 great pieces each month. That’s, uh, $2,000 per item if you do the simplest math (and everyone will). Yikes.

But that one piece that got into Overlord Weekly generated 25 great links, at least 1,000 new followers, and $5,000 in sales. Compared to the banner ad campaign, or the direct mail campaign, that’s a fantastic result. Be sure your boss or client knows.

Most important: Stick with it. A dashboard with gaps and inconsistent data is as believable as Nice Smeagol. Keep it in mind.

The post SEO Analytics, Middle Earth-Style appeared first on Portent.

Ask your own question in the comments. Depending on how many questions I receive I’ll answer either all or the best ones in future blog posts.

How many PPC keywords should you buy?

Continuing the Portent PPC series, today’s insight is how to determine how many keywords you should have in any given ad group or campaign.

Ultimately, the actual number of appropriate keywords depends on your industry, but the rule of thumb is keep it simple and group PPC keywords related to your business appropriately.

One sure sign: if you have to run an Adwords Search Query Report every time you log into your Google account you’re most likely bidding on too many terms (unless you are Target or Amazon).  If it’s in the hundreds, it’s too many.

Too many keywords!

High volume keywords can create several thousand impressions each day. Break them out into their own campaigns to control costs and ease maintenance. Some examples are real estate, lawyer, cars, jewelry, bridal, furniture, and shoes.

If you must bid on a difficult to control PPC keyword, place it in a separate campaign. Make sure that you actually need that keyword and that it converts. This way you can control costs, exposure and filter out low quality clicks and visits. Ideally, you want to have no more than 20-25 keywords in an ad group. This will make it easier to not only see all the keywords on a single page in the PPC search engine’s interface, but also keep things neat and increase quality score potential.

The post PPC FAQ – Finding the Right Number of Keywords appeared first on Portent.

How Do I Find Great PPC keywords?

Know what you are selling. Bid on your products and services and avoid buying keywords for products or services you do not carry.

Most search engines offer a free keyword tool to help you generate keywords. In some cases these will estimate search volume for new keywords.

You can generate additional keywords by

• current keyword list
• keywords you type in
• URL submission
• topic or category

Begin by using the Google AdWords: Keyword Tool to create a keyword universe then cut away PPC keywords with no search volume or queries that will not work for your company.

1. Begin with an accurate, basic search query and sort the results by relevance.
2. Take the most relevant results and run Google Search queries on them.
3. Research the top web site results in the Google AdWords: Keyword Tool.
4. Keep repeating the process with new search queries and new web sites until you only get repetitions.
5. Throughout your research keep exporting the results as CSV files. Later you can merge these and sort the results to uncover the most popular, most relevant, most expensive, and bargain PPC keywords for your business.

Keep in mind that a shotgun approach will cost you dollars. You become less relevant in the eyes of the search engine algorithms, your web site will get penalized and you will pay a elevated cost per click prices.

Do not bid on every possible keyword. This creates a horrible user experience. If someone searching for “women’s sandals” arrives on your web site and discovers that you only sell sneakers they will leave angry likely never return.

Finally, you may wish to use a pay service to acquire alternative spellings, related keywords and predicted search volume.

The post PPC FAQ – Action Packed PPC Keyword Research appeared first on Portent.