collection, the web crawler downloads and replaces the web page in the collection with a first probability; and when the web page is determined not to be present in the collection, the web crawler downloads and including the web page in the collection. 31.