![[Home]](http://meatballwiki.org/meatball.gif)
SpiderTrap
MeatballWiki | RecentChanges | Random Page | Indices | CategoriesQuite often for large sites like wikis that are generated dynamically, or for an OnlineCommunity that wants to remain more or less as a HiddenCommunity?, WebCrawlers (aka spiders) are a big problem. While the RobotsExclusionStandard is supposed to protect a site against wayward spiders, not all spiders adhere to this standard, particularly those searching for e-mail addresses to harvest for spam. Further, the RobotsExclusionStandard a very binary response: spiders are only allowed in or excluded altogether. Crafting a response particular to spiders is not an option.
To solve these problems, one needs to first detect them in order to respond. If you note, spiders cannot read semantically. They only read links, and then follow links. If you present a link that only a spider will follow, hopefully the spider will follow the link and your users won't. In this way, it is machine-oriented fly-paper.
Ways of presenting the link so that users won't click on it but spiders will are various.
- Write a warning. Make the link text explicitly say to people, "Clicking on this link will lock you out of the system." Of course, with all things that say "Do not touch this," people will be drawn to touch it. (cf. LimitTemptation)
- Commented out. Bury the link in a comment.
<!-- <a href="http://...">spider trap</a> -->. This won't display to an end user, but for dumb spiders that just search for href=, this will get them.
- No text. Don't have any link text. Most browsers will present nothing to the user to click on.
<a href="http://..."></a>
- Small text. Make the link text very small so users will have a hard time finding it.
<a href="http://..."> </a>makes the link text only a non-breaking space.
- White text. Make the text the same colour as the background.
<a style="color:white" href="http://..."> </a>
It should be noted that Google hates people who hides links deviously like this on their sites. The simplest and best solution may simply be - a la HidingInPlainSight? and OpenProcess - to write a link <a href="http://...">Spider trap</a>, but then give curious cats a way out of the mess they have entered. On the next screen (or all the screens thereafter), offer opportunities for HumanVerification (e.g. a CAPTCHA).
Alternatively, there are SpiderTraps designed for spam-bots which need not be hidden from users, such as a SpamPoisoner.
You can make a SpiderTrap link a SelfBan as well, if you want to block all spiders. This is somewhat dangerous as unwitting users will routinely click on the URL, and it exposes the URL to would be attackers. If you do this, you may have to use some extra precautions. You could generate a nonce every time you display the SpiderTrap-SelfBan link, and expire it quickly. Another option is to allow people to unban themselves with HumanVerification.
You can combine the SpiderTrap with a SurgeProtector. If the spider trips the trap more than X times a minute, or X percentage of total hits, you block it then. This can help detect misbehaved spiders (like a wget) without banning GoogleBot? or Yahoo! Slurp. Additionally, this can help overcome the problem of users self-banning themselves. If someone is dumb enough to continuously load the SpiderTrap, perhaps they deserve being banned.
Warning. SearchEngineCloak detection algorithms are secret. They simply use a second spider to check to see the pages are similar. If you SpiderTrap one and not the other, your site may be banned. The solution is to put the SpiderTrap link in RobotsDotTxt? - after all, you are looking for bots that do not adhere to the RobotsExclusionStandard.
CategoryHardSecurity CategorySpam