Let me explain a bit.
To exclude all robots, that respect the robots.txt file: User-agent: *
Disallow: / To exclude just one directory and its subdirectories, say, the /aggregator/ directory: User-agent: *
Disallow: /aggregator/ To disallow specific robots you need to know what it calls itself, ia_archiver is the wayback machine
To allow the Internet Archive bot you'd make a line like this: User-agent: ia_archiver
Disallow: To block ia_archiver from visiting: User-agent: ia_archiver
Disallow: /
You can have as many lines like this as you want. So you can disallow all robots from everywhere, and then allow only those you want. You can block certain robots from certain parts. You can block directories and sub directories or individual files.. If you have numerous "aggregator" files in various subdirectories you want to block you need to list them all.
Like this:
User-agent: * Disallow:/aggregator/ Disallow:/foo/aggretator/ ...
Disallow:/hidden/aggregator/
Your syntax looks wonky, missing the final "/". User-agent tells who to block and Disallow what to block. This all assumes well behaved robots. This file is useless for those that ignore this file. It is not a security device, just a polite sticky note.
You might go here for more detailed info. I'm no expert for sure.
http://www.robotstxt.org/orig.html
Jack
--- On Sun, 1/16/11, Jonathan Hutchins [email protected] wrote:
From: Jonathan Hutchins [email protected] Subject: robots.txt question To: "KCLUG (E-mail)" [email protected] Date: Sunday, January 16, 2011, 12:53 PM
I'm wondering about the syntax. The example file from drupal uses the format
Disallow: /aggregator
However, it says in the comments that only the root /robots.txt file is valid.
From my understanding of the syntax, /aggregator does not
block /foo/aggregator, so I need to either prepend "/foo" to everything, or use wildcards per the new google/webcrawler extensions to the protocol.
If anybody can cite an on-line example that explains I'd be grateful.