"The robots.txt file is a file you can place on your server to specify how automated software should be allowed to access your pages." . . . .