| Spider Trap |
Article Index for Spider |
Website Links For Spider |
Information AboutSpider Trap |
|
Common techniques used are:
There is no algorithm to detect all spider traps. Some classes of traps can be detected automatically, but new, unrecognized traps arise quickly. POLITENESS A spider trap causes a web crawler to enter something like an Infinite Loop , which wastes the spider's resources, lowers its productivity, and, in the case of a poorly written crawler, can crash the program. Polite spiders alternate requests between different hosts, and don't request documents from the same server more than once every several seconds, meaning that a "polite" web crawler is affected to a much lesser degree than an "impolite" crawler. SEE ALSO
EXTERNAL LINKS
|
|
|