Moreover, with the advent of the CommonGatewayInterface? (CGI), many portions of a website may not be appropriate for a spider, either because the CGI script may take a lot of CPU time (in the case of wikis certainly) or because the script tracks user information, e.g. the ShoppingCart? sections of amazon.com.
There are many other cases where it is important to block robots and indexers from your site.
The RES provides two mechanisms to exclude robots: robots.txt and the Robots meta tag. These are both detailed in the above link.
One problem with this, if used injudiciously, is that you end up telling everyone where all the good stuff is. Not good. Of course, enough people have been burned that hackers know to look there and webmasters mostly know to handle it tenderly. For example, don't put "disallow:/hidden_creditcard_numbers/", put the credit card numbers in /Data/something/creditcard_numbers/ and make "something" have few permissions and not give any information or subdirectories. Do the same with Data, and preferably have a password-like name for "something" (non-obvious, not a real word)
Passwords are also SecurityThroughObscurity by this logic, yet oddly they're still widely used; I don't think your sentiment, "[this] is considered to be no security at all", is as authoritative as it sounds. Nevertheless, I agree that credit card numbers are best protected by hard security. In this case, that doesn't mean faffing around with access permissions: it means taking them out of public_html. Trivial. -- ChrisPurcell