Gifted Typist

Phew! Just taking a break after a tough few days of fighing fires here at GT. This site has been under attack by Bad Bots.

The other morning I checked my stats to find more than 500 hits from the same IP address. In two hours! It’s hard work trying to track them down , research their sources and figuring out what to do. And I wasn’t finished fighting one fire when another one started and another one again. Almost 800 in a day. Help!

Bad bots are different things. Many are spiders that crawl your site to harvest market information and other data like email address. I’ve been reliably informed never to include an email address in your blog.

A site ripper or scraper will take your content and download it for offline use. Sometimes the content is used somewhere else as a front for Google Ads and other banners.

The problem is that these crawlers drive up your bandwidth usage sometimes to the point of crashing your server. Bad bots typically ignore the wishes of your robots.txt file, so you’ll want to ban them using means such as .htaccess. If you know what THAT is.

The trick is to identify a bad bot. Here is a useful site to deal with these problems for anyone interested.

This entry was posted on Sunday, May 20th, 2007 at 1:05 pm and is filed under blogs. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.