Wednesday, April 09, 2008

Blocking bad bots

Today I blocked some bad bots that were spidering some of my sites. Most notably Custo, which downloads your entire site.

An interesting solution is posted here (I used the mod_rewrite option). You can test this by changing your user agent in Firefox.

This guy seems to be following bad bots.

I added Java, Nutch, Jakarta, Vagabondo and an empty bot name to the list of bad bots.

Labels: ,

0 Comments:

Post a Comment

Links to this post:

Create a Link

<< Home