While the robots.txt file was invented to inform search engines like yahoo what web pages to not crawl, the robots.txt file can even be accustomed to stage search engines like google into the XML sitemap. This can be supported by Google, Bing, Yahoo and Inquire. The Crawl-delay directive is an https://rankingseotoolz.com/robots-txt-generator