Interessante robots.txt
Verfasst: 08.07.2004, 12:34
Bin auf eine Webseite gestossen, die folgende robots.txt hat :
# This robots.txt file is generated based on the IP Address and
# User Agent requesting it. We grant broad access to certain
# IPs and Agents -- mostly known search engine spiders. All
# others are sent the default rules denying access to the entire
# site. If you have a need to access our site via robots.txt,
# please send an explain by email to our webmaster and we will
# consider granting your IP Address and User Agent an appropriate
# level of access.
#
# Thank you for your interest.
#
User-agent: *
Disallow: /
Also robots.txt cloaking
Hat aber den Vorteil, dass man eine Whitelist führen kann, statt einer Blacklist, die ständig aktualisiert werden muss.
Gruss Gunnar
# This robots.txt file is generated based on the IP Address and
# User Agent requesting it. We grant broad access to certain
# IPs and Agents -- mostly known search engine spiders. All
# others are sent the default rules denying access to the entire
# site. If you have a need to access our site via robots.txt,
# please send an explain by email to our webmaster and we will
# consider granting your IP Address and User Agent an appropriate
# level of access.
#
# Thank you for your interest.
#
User-agent: *
Disallow: /
Also robots.txt cloaking
Hat aber den Vorteil, dass man eine Whitelist führen kann, statt einer Blacklist, die ständig aktualisiert werden muss.
Gruss Gunnar
