THE CURRENT VERSION IS SEY 2003

How Search Engines Work

Let’s start by distinguishing between search engines and directories.

Search Engines (like www.google.com)

The main characteristic of search engines is that they rely on spiders to crawl the web, indexing pages as they go. Spiders are browser-like programs that follow links from page to page and from site to site, indexing everything it finds.

When you submit a web page to a search engine, all you really do is tell the spider about the page.Your page does not get added to the search engine’s database immediately – that only happens once the spider gets around to visiting and indexing the page.

There are very few real search engines.

You'll often see claims like "We'll submit your site to 10,000 search engines". That's nonsense. There simply are not 10,000 or even 1,000 real search engines. There are maybe 100 worth knowing about and no more than 10 worth submitting to.

People who make claims like the one above often submit your site to thousands of FFA pages and obscure directories. Submissions to FFA pages could actually hurt your site.

For more specific information on Google, please refer to the Google section of this site.

Directories (like dmoz.org)

Directories do not use spiders.

Instead, they use real people (editors) who visit and evaluate sites – and add them only if they meet the directory’s minimum quality requirement.

This is an important difference:

• Search engine spiders can index thousands of pages a day.
• Directory editors cannot.

So why do we have directories if they can’t compete?

The answer is quality.

Editors are considerably harder to impress than spiders. The page has to offer unique information or a unique product. When you submit a page to a specific category in a directory, the editor of that category will visit your page and decide if it’s good enough to add to the directory.

Editors usually reject pages with typos, broken links, unclear navigation etc.

For more on submitting to DMOZ specifically, refer to Getting Listed at DMOZ (ODP)

The Components Of Search Engines

Search engines and directories all consist of 5 major components:
1. The spider (or editor in the case of directories)
2. The indexer (again the editor in the case of directories)
3. The database
4. The search software
5. The interface

For a more detailed explanation of what each component does as well as how they fit together, please refer to the current version of the Search Engine Yearbook.