Inbook,

Crawling the Web

, , and .
page 153--177. Springer Berlin Heidelberg, Berlin, Heidelberg, (2004)
DOI: 10.1007/978-3-662-10874-1_7

Abstract

The large size and the dynamic nature of the Web make it necessary to continually maintain Web based information retrieval systems. Crawlers facilitate this process by following hyperlinks in Web pages to automatically download new and updated Web pages. While some systems rely on crawlers that exhaustively crawl the Web, others incorporate ``focus'' within their crawlers to harvest application- or topic-specific collections. In this chapter we discuss the basic issues related to developing an infrastructure for crawlers. This is followed by a review of several topical crawling algorithms, and evaluation metrics that may be used to judge their performance. Given that many innovative applications of Web crawling are still being invented, we briefly discuss some that have already been developed.

Tags

Users

  • @dret
  • @fil
  • @parismic
  • @dblp

Comments and Reviews