-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, May 10, 2007 at 02:27:39AM +1000, Tony Smith wrote:
> Some sites serve up different content depending on who's visiting. 
> 
> At lot of it is no big deal, media sites lock out non-US visitors, some
> switch languages based on IP and then there are changes made depending on
> your browser.
> 
> Naturally, it's not hard to spot the Google spider, and give it access to an
> abridged PDF (ya don't want the lot ending up in the cache!).  AKA 'doorway'
> pages.  Google takes a dim view of this, and banishes those it catches.  I'm
> not sure how it catches them though, a second spider disguised as a browser?

It really must be.

Search for any of my pages, and a decent number will have weird crap in
the google results from my "put up the webservers logs" background.

For awhile I had some code that would detect the google spider and
simply disable that stuff. But I noticed that every new page I put up
would work... then about a week or two later that log crap would show up
again.

Google definetely has second spiders.

> Flash sites are worse, they never show up.  Then again, they don't deserve
> to.

I agree %100...

Fortunately lots of artists in competition with me have flash sites...
I can think of a few very well known ones who don't even show up.

- -- 
http://petertodd.ca
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFGQgvE3bMhDbI9xWQRAteZAKCiFf5hCofT6gBN43d9UgrvcxMzdQCfVoYT
jmqCgmjrCf9s/Zh2dkPgVe0=
=ZBLg
-----END PGP SIGNATURE-----
-- 
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist