#Default robots.txt file (ask all bots to crawl slowly but still index everything). User-agent: * Crawl-delay: 1 Disallow: