. "Robots.txt is a file which is placed on a website to guide indexing algorithms (crawlers.) Its syntax is defined as a standard (Robots Exclusion Standard) and pretty much every search engine crawler obeys it." . . .