DerEinzige & traumfahne.de
zeitschrift zum selbstausdrucken im eigenverlag vom 26. 3. 2007 | gesamtausgaben: 197
robots database
26. März 2007 | ausgabe drucken | Kommentare deaktiviertleserbrief schreibenhaving a rather long list of searchbots, crawlers, spiders and grabbers, i decided to put them in a xml database for common share:
sort order: first the allowed, the wildcard, then the disallowed. both alphbetical ascending.
when you take a look, you will spot it is raw and inkomlete. i promise, i’ll work on it every now and then 
most of the urls are missing, notes on the bots are rare … i would appreciate any kind of help expanding and completing the database … (please use the comment form or drop me a line)
also of interest are bots who have such commands like delay … slurp and irlbot have … two in 366 
witch are the real spambots? witch don’t obey the robots.txt …
autor: jeff brett, verfasst am 26. 3. 2007
permalink: http://dereinzige.de/2007/03/26/robots-database/
short url: http://dereinzige.de/204
