All About Search Indexing Robots and Spiders
   Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.
   http://www.searchtools.com/robots/

Bots vs Browsers
   This large database lists user agents in categories and distinguishes between robots and browsers.
   http://www.botsvsbrowsers.com

HTTP User Agent Index
   An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.
   http://www.siteware.ch/webresources/useragents/db.html

List of Robot Agent Strings
   A list from PGTS of Web robots with the identifying data they leave in Web site logs.
   http://www.pgts.com.au/pgtsj/pgtsj0208d.html

Robot IP Address
   Brian Dunnintg provides a list of all the major search engine robot IP addresses, by full class C only.
   http://www.briandunning.com/seo/

Robotstxt.org
   Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.
   http://www.robotstxt.org/

Search Engine IP Addresses
   Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.
   http://www.iplists.com/

Search Engine Robots and Other User Agents
   John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.
   http://www.jafsoft.com/searchengines/webbots.html

Spider Track
   Tracks and displays real time hits and IP addresses of MSNbot, Yahoo's Slurp, and Googlebot. Provides aggregate hits of each spider over the last 30 days.
   http://www.spidertrack.org

User Agent String
   Tool from ASAP Consulting s.r.o. for detailed user agent string analysis using an online form. Includes databases of browsers and robots.
   http://user-agent-string.info/

User-Agents.org
   Large list of search engine spiders, similar web robots, and Web browsers: their web-log identification and links to their originators.
   http://www.user-agents.org/