Nate Duehr wrote:
> James Newton, Host wrote:
>
>   
>> BTW: So far this month, the site is averaging 68,641 hits per day, which is
>> up 37%. Good thing I have google as my hosting bill will be higher again.
>>     
>
> How many of those are bots?  Do you reverse-resolve incoming HTTP 
> requests?
>
> Many of the bots and crawlers at least identify themselves in HTTP 
> request headers and/or by their reverse DNS entries.
>
> I have "thousands" of hits to my website in my Apache logs, but easily 
> 30% of it is the crawlers and people caching BackTrace info from other 
> people's "blogs" (what a dumb word).
>
> Since the information is meant to be "public" I don't block them or 
> place a robots.txt file on the site that's restrictive unless things get 
> totally out of hand...
>
> Nate
>   
I think things are really out of hand if the server crashes from it, 
maybe just put a robots.txt which prevents getting the large pdf's from 
webcrawlers

-- 
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist