I was surfing around and ran across this site. It allows you to add text into your html that disallows some of the spider features you might not want .
Also it will disallow those nasty email grabbers (some of them)
Anybody every used it before ? I want to apply it to my most famous blog! But, I also don’t want it to kill any chances of keeping it on the search engines.. There are a few other sites it might be good for also ..




Example code disallowing NASTY robots to spider your site

#
# robots.txt generated by www.1-hit.com's robot generator
# Please, we do NOT allow nonauthorized robots any longer.
#

User-agent: asterias
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: Black Hole
Disallow: /
User-agent: BlowFish/1.0
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye/1.0
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: Cegbfeieh
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: CherryPickerElite/1.0
Disallow: /
User-agent: CherryPickerSE/1.0
Disallow: /
User-agent: CopyRightCheck

I don't see how that would kill search engines like Google, Yahoo, MSN, etc... since their bots aren't even listed.

So then you like it? You aggree with using it?

Just remember, your robots.txt file is a suggestion to bots that just asks, "please don't look at this part of my web site!" The really nasty robots won't check robots.txt, simply ignore it, or worse -- concentrate on the areas that are off-limits.

BTW - why does the web site list Mozilla and Wget as nasty bots?


This post has been edited by Moldz: May 23 2006, 07:33 AM

I dont know why it listed Mozilla and Wget ? Seems like those are kind of passive folks

I read an interesting article sometime last year or so about punishing the spiders that didn't obey the robots file. They basically made a tarpit with some code that bogged the spider down, making it think that it was caching thousands of files instead of the handful that kept renaming themselves. That area of the site was then added to the robots file and the good bots would avoid it while the bad bots would get stuck there for hours. I think somebody here may have done that to their site.

http://computerhelpforum.org/forum/programming/f46/http_computerhelpforum_org_forum_index_php_showtopic_6186/t6186.html

That's not what I'm talking about, but it's kinda close. That's the "generate fake files to screw with the RIAA" thing. The one I can remember tried to keep the spider in an endless loop.