ON 20040326@11:09:07 AM at page:
http://techref.massmind.org/techref/inet/wwws.htm#38072.4646527778
James Newton[JMN-EFP-786] Says
http://techref.massmind.org/techref/inet/robots.htm
Robots.txt lists items that (well behaved) index engines (search bots, web crawlers, etc...) should not request.



ON 20040326@11:11:50 AM at page:
http://techref.massmind.org/techref/inet/robots.htm#38072.4665509259
James Newton[JMN-EFP-786] See also:
http://www.robotstxt.org/wc/norobots.html
<h2><a name="format">The Format</a></h2>

The format and semantics of the "<code>/robots.txt</code>" file
are as follows:

<p>

The file consists of one or more records separated by one or
more blank lines (terminated by CR,CR/NL, or NL). Each
record contains lines of the form
"<code>&lt;field&gt;:&lt;optionalspace&gt;&lt;value&gt;&lt;optionalspace&gt;</code>".
The field name is case insensitive.</p>

<p>

Comments can be included in file using UNIX bourne shell
conventions: the '<code>#</code>' character is used to
indicate that preceding space (if any) and the remainder of
the line up to the line termination is discarded.
Lines containing only a comment are discarded completely,
and therefore do not indicate a record boundary.</p>

<p>
The record starts with one or more <code>User-agent</code>
lines, followed by one or more <code>Disallow</code> lines,
as detailed below. Unrecognised headers are ignored.</p>

<dl>
<dt>User-agent</dt>
<dd>

The value of this field is the name of the robot the
record is describing access policy for.

<p>
If more than one User-agent field is present the record
describes an identical access policy for more
than one robot. At least one field needs to be present
per record.</p>

<p>
The robot should be liberal in interpreting this field.
A case insensitive substring match of the name without
version information is recommended.</p>

<p>

If the value is '<code>*</code>', the record describes
the default access policy for any robot that has not
matched any of the other records. It is not allowed to
have multiple such records in the "<code>/robots.txt</code>"
file.</p></dd>

<dt>Disallow</dt>
<dd>

The value of this field specifies a partial URL that is not
to be visited. This can be a full path, or a partial
path; any URL that starts with this value will not be
retrieved. For example, <code>Disallow: /help</code>
disallows both <code>/help.html</code> and
<code>/help/index.html</code>, whereas
<code>Disallow: /help/</code> would disallow
<code>/help/index.html</code>
but allow <code>/help.html</code>.

<p>

Any empty value, indicates that all URLs can be
retrieved. At least one Disallow field needs to
be present in a record.</p></dd>

</dl>

The presence of an empty "<code>/robots.txt</code>" file
has no explicit associated semantics, it will be treated
as if it was not present, i.e. all robots will consider
themselves welcome.

<hr>

<h2><a name="examples">Examples</a></h2>

The following example "<code>/robots.txt</code>" file specifies
that no robots should visit any URL starting with
"<code>/cyberworld/map/</code>" or
"<code>/tmp/</code>", or <code>/foo.html</code>:

<hr>
<pre>
# robots.txt for http://www.example.com/

User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html
</pre>
<hr>

This example "<code>/robots.txt</code>" file specifies
that no robots should visit any URL starting with
"<code>/cyberworld/map/</code>", except the robot called
"<code>cybermapper</code>":

<hr>
<pre>
# robots.txt for http://www.example.com/

User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space

# Cybermapper knows where to go.
User-agent: cybermapper
Disallow:
</pre>
<hr>

This example indicates that no robots should visit
this site further:

<hr>
<pre>
# go away
User-agent: *
Disallow: /
</pre>
<hr>