Wots the ip and can we all play with it please ???

Steve

-----Original Message-----
From: piclist-bounces@mit.edu [mailto:piclist-bounces@mit.edu] On Behalf Of
Harold Hallikainen
Sent: 25 May 2006 19:20
To: Microcontroller discussion list - Public.
Subject: Re: [OT] PicList Traffic

I had so many search engine crawlers going through my site that they
brought down the server when lots of them downloaded large pdfs. This
first happened, of course, when I was out of town for a week and could not
bring the server back up. Since then I've added a remote power on/off
device with an embedded web server. So, worst case, I can power cycle the
machine. A few things I did to fix the problem were:

1. Add a cron job that restarts httpd if the load exceeds 13 or whatever
it is when sendmail stops accepting mail. This script also sends me an
email telling me it did this.

2. Add crawl-delay to my robots.txt file. This backed the robots down from
every second or so to once a minute.

3. Added stuff to crawl-delay that keeps robots from downloading those big
pdfs.

Doing items 2 and 3 has kept item 1 from happening for the past two months.

Harold


> James Newton, Host wrote:
>
>> BTW: So far this month, the site is averaging 68,641 hits per day, which
>> is
>> up 37%. Good thing I have google as my hosting bill will be higher
>> again.
>
> How many of those are bots?  Do you reverse-resolve incoming HTTP
> requests?
>
> Many of the bots and crawlers at least identify themselves in HTTP
> request headers and/or by their reverse DNS entries.
>
> I have "thousands" of hits to my website in my Apache logs, but easily
> 30% of it is the crawlers and people caching BackTrace info from other
> people's "blogs" (what a dumb word).
>
> Since the information is meant to be "public" I don't block them or
> place a robots.txt file on the site that's restrictive unless things get
> totally out of hand...
>
> Nate
> --
> http://www.piclist.com PIC/SX FAQ & list archive
> View/change your membership options at
> http://mailman.mit.edu/mailman/listinfo/piclist
>


-- 
FCC Rules Updated Daily at http://www.hallikainen.com - Advertising
opportunities available!
-- 
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist


-- 
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist