-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, May 11, 2007 at 01:18:12AM +1000, Tony Smith wrote: > > It really must be. > > > > Search for any of my pages, and a decent number will have > > weird crap in the google results from my "put up the > > webservers logs" background. > > > > For awhile I had some code that would detect the google > > spider and simply disable that stuff. But I noticed that > > every new page I put up would work... then about a week or > > two later that log crap would show up again. > > > > Google definetely has second spiders. > > > Why don't you use robots.txt like you're supposed to? No, the situation is more complex than that. See, basically I'm presenting a page that google can't understand very well. That's because each page has, for purely asthetic reasons, a whole pile of server logs. Google thinks that text is important, istead of decoration, and shows it on search results. In a sense it is. I get a *lot* of people coming to my webpage, like 50 or so per day, from google searches that match the zillions of worms and stuff trying to break into my system. Heck, I sometimes get calls from very confused sysadmins who somehow think that because the worm's IP shows up on my server logs too I'm trying to hack into their system. > That's exactly the sort of thing that gets you kicked out of Google. > Serving up a different result to the Google spider than what a browser would > see means you're trying to rig the system. Browsers see spam, spider sees > keywords. Tsk, naughty! Only in this case I'm actually trying to *prevent* google from seeing keywords. > Anyway, I doubt the spider runs Javascript, so it may not have even noticed > unless you were doing it server-side. Which is exactly it... I do do it server-side, with a very simple php line that's literally: But doing it with javascript is really a very good idea... - -- http://petertodd.ca -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD4DBQFGQ3O9pEFN739thowRAmuGAJ9kvOUMZ7xqSSuOzOyQYzfan2F+YQCWJLCd z4UxXeL668yY/DRI7haQYQ== =pK7w -----END PGP SIGNATURE----- -- http://www.piclist.com PIC/SX FAQ & list archive View/change your membership options at http://mailman.mit.edu/mailman/listinfo/piclist