RSS 2.0 Feed

Death of an index

Posted on 21st December 2007 at 2:56 am by Manley

Now that its supplemental results will no longer be . . . well, supplemental, I suppose . . . where does that leave those pages which were languishing in Google purgatory?

Back in 2003, Google's index was simply getting too large. The index wars were not to end until Google removed the number of pages indexed from its home page, some two years later, and the resource required to process all the data meant annexing some of the index off.

This auxiliary index was referenced for search terms with fewer results and allowed pages to be indexed which were not meeting the constraints required by the main index.

Google named this the supplemental index and clearly marked those results which were drawn from the secondary index accordingly:

Image showing supplemental results on a hypothetical Google results page

Then, in July 2007, Google stopped labelling supplemental results. Ostensibly this was because the difference between the two indices was narrowing. More pages could be included within the main index, URLs with more parameters were being effectively crawled and Google was promising to work on including both indices in every query. However there was an additional theory at the time that Google wanted webmasters to stop focusing on small details and to put more effort into developing interesting, value-adding content.

There were still ways for a search professionals to determine which index a page was in. Over time some of these search modifiers stopped working and new ones were found, but still the divide narrowed and, on Tuesday 18th December, Google announced that it now searches the whole index for every query.

Whether this means that there is only one index or whether, as I suspect, there remain two separate indices but with all artificial divides removed, is largely irrelevant. What matters is how this will affect end users and webmasters.

Google says that, from a user perspective, this means more relevant documents and a vaster pool of resource. Particular reference is made to non-English queries (international clients have historically suffered from lack of exposure in the main index). For webmasters, Google says that this means that good-quality pages that were less visible in its index are now more likely to come up for queries.

I doubt that the two indices are weighted equally although - with it being increasingly difficult to ascertain which index a page is in as this change is rolled out across datacenters, it is almost impossible to determine how results from one index are performing against the other - but a deeper, more comprehensive search should mean a boost for less visible, good quality pages which target long-tail queries.

File under: google supplemental