Even a medium-sized site may take hours or days to fully crawl when rate-limiting and other factors are taken into account, and for some users it is imperative to know the precise moment when each page was requested, not when the crawlers first entered the site.