It should be clear, however, that it is not essential to the invention that this hardware and operating system be used, and other hardware and operating systems can be used such as UNIX or Microsoft Windows XP. In an exemplary embodiment, multiple instances of the crawler 102 run to increase capacity to retrieve hypertext document collections such as web pages on the Web 101.