One method of measuring the SEO health of your web site is determining if your most important web pages like i.e the most lucrative products or services have been moved to Google’s supplemental index or not.
Many people believe that since Google stopped labelling the supplemental results pages, that the supplemental index has gone. That is not true!
Google clearly stated in their article "Supplemental goes mainstream" at their Webmaster Central Blog:
"The distinction between the main and the supplemental index is therefore continuing to narrow. Given all the progress that we’ve been able to make so far, and thinking ahead to future improvements, we’ve decided to stop labeling these URLs as Supplemental Results. Of course, you will continue to benefit from Google’s supplemental index being deeper and fresher."
The pages that show up first for any searches are the ones in the main index. The only time you will see pages from the supplemental index, is if there are not many, or any results for that term in the main index.
Also Google tends to move old cached pages to their supplemental index, which can also be pages that you do not even have on your server anymore.
Bot Herding for PageRank Flow
To be in Google’s main index, your pages need to have a certain undefined amount of juice (Pagerank), independent of other factors being involved. Google uses PageRank values to set crawling priorities and to determine if a document should go into their main or supplemental index.
Matt Cutts, head of the Google Webspam Team’s quote, speaks for itself:
"PageRank is the primary factor determining whether a url is in the main web index vs. the supplemental results."
Once you get to know the common causes of supplemental pages you can determine which pages may be in the supplemental index. Then you can improve your web site internal linking by adding links to these pages from more prominent, fully indexed pages of your site eg. your homepage.
Thanks to Andy Beal, you can hear Matt Cutts say basically the same thing again:
"If you got 60,000 pages, and you only got ’this much’ PageRank, and you divide it [...he mumbles], some of them are going to be in the supplemental index. Given 'this many people' who link to you, we’re willing to include 'this many' pages in the main index."
An SEO or a Link Builder would probably advise you that the best way to get your pages out of the supplemental results is to create high quality, unique content and promote it to get inbound links. That is correct. But, is that really the best route to take? Why not first try to see how far you can get with the PageRank that your web site already has?
There are several internal link-based tactics at your disposal to combat supplemental results. One popular and effective strategy is commonly known as Bot Herding.
Bot Herding is nothing more than a methodology to improve your web site navigation system by controlling the flow of PageRank to boost the prominence of your most important web pages. This can be achieved by linking to them from pages within your domain, and more.
Common causes of supplemental pages
The most common cause of Supplemental results is lack of Page Rank. Still there are some more causes which must be taken into account:
- Pages due to canonicalization problems, e.g duplicated content, too much content similarity;
- Pages with too little or no original content;
- Orphaned web pages. Pages that no one links to, including your own;
- Error pages, if a site does not use If-Modified-Since, Last Modified and/or Expires rules;
- Poor website navigation;
- Loading pages with irrelevant keywords (keyword stuffing);
- Pages with no or too low PageRank;
- Long URLs, especially with long parameters, starting with a question mark (?) and being separated with an ampersand (&) and are not rewritten;
- Suspicious pages for spamdexing, like non-unique and irrelevant to page content heading tags, meta tags, or linking to bad neighborhoods, etc.
Do you have pages in the supplemental index? Get them out of there, save time and money, and boost your rankings!
About this SEO article
This tutorial was written by John S. Britsios (aka Webnauts), Search Experience Consultant at SEO Workers and was published September 12, 2007.
Copyright reserved. Not to be reproduced.