Peter P. wrote:

> It isn't just that they are not cached, Google *knows* that it should not
> cache them as there is no link to 'view as HTML'. There must be
> something else going on there. And it's a something that has a strong
> flavor of unequal treatment to me. Then there's Ebay pages which appear
> on Google a few hours after an item is put up. Contrast this to the
> shortest time it took (ever) to get a fairly public site noticed and
> ranked. Oh, and the Ebay pages appear very near the top of the link
> listing (in the 1st page at least). There are several 'strange' things
> like this out there, all connected to 'brand names'. 

A few years ago I vaguely noticed that the Google ranking lost a lot of the
up to then quite astonishing usefulness for me, and this trend has itself
confirmed over the years. This could well be related.

FWIW, the robots.txt file at http://linkinghub.elsevier.com goes like this:

  # /robots.txt file for http://linkinghub.elsevier.com/
  User-agent: *
  Disallow: /

No crawling at all...

The sciencedirect.com site allows Googlebot (among others), but that's not
the link listed in Google. It looks like elsevier.com made the content
available to Google for indexing, and probably not through the normal bot
access.

Gerhard

-- 
http://www.piclist.com PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist