THE CURRENT VERSION IS SEY 2003
All About Robots
& Spiders
by
André le Roux
Dec. 2002
What's a "Robot" And A "Spider"? What's The Difference?
Robots - in the context of the Internet - are browser-like programs that automatically surf the web. They do this by following links from one page to the next and from one site to the next.
Robots have many different uses. Some collect information on web sites, some harvest e-mail addresses, some check link validity etc.
Spiders are also robots, but the term is used to specifically refer to robots used by search engines to collect information about web sites. Spiders "index" pages they find - in other words they record the information on these pages. This information is dumped in a database. When you do a search at a search engine, it queries the database (also called the index) rather than really "searching the web".
Spider Spotting
For various reasons, webmasters often track spider activity on their sites. You may have noticed names like "Scooter" and "Googlebot" in your server log files. These are spider names. Here are a couple of names you are likely to encounter:
| Search engine | Spider name |
| Abacho | AbachoBOT |
| Aesop | AESOP_com_SpiderMan |
| Ah-ha | ah-ha.com crawler |
| Alexa | ia_archiver |
| AltaVista | Scooter |
| AlltheWeb | FAST-WebCrawler |
| Atomz | Atomz |
| Excite | ArchitextSpider |
| Euroseek | Arachnoidea |
| EZResults | EZResult |
| Googlebot | |
| Inktomi | Slurp |
| Lexis-Nexis | LNSpiderguy |
| LookSmart | MantraAgent |
| Lycos | Lycos_Spider_(T-Rex) |
| Mirago | HenryTheMiragoRobot |
| Northernlight | Gulliver |
| National Directory | NationalDirectory-SuperSpider |
| Openfind | Openfind piranha,Shark |
| SearchHippo | Fluffy the spider |
| Teoma | teoma_agent1 |
| Ttravel | Finder ESISmartSpider |
| UKSearcher | UK Searcher Spider |
| Walhello | appie |
| Websmostlinked | Nazilla |
| Wisenut | ZyBorg |
For more information on controlling spider activity
on your web site, please refer to the Robots.txt
section of this site.
This
page is based on information contained in the Search Engine Yearbook 2003.
For more detailed search engine information & help, please refer to the
current version of the book.
Stay
up to date on changes in the search engine world with the EnginePaper
Newsletter. It goes out only when something important changes in
the search engine world. Subscribe now with a blank email to
send-ep-subscribe@topica.com
. It's 100% free and safe. View our full privacy policy
here.