Overview Robots

What is a robots.txt file

Search engine spiders check the root of your website directory for a robots.txt file. In the robots.txt are directives for search engine spiders what they are allowed to index and what you don’t want to get indexed by the search engine spiders. You can specify per spider what is allowed and what is not allowed (disallowed). If you have a sitemap you can specify it’s location in the robots.txt file.

Do i need a robots.txt file

If you want to control what search engine spiders index from your website you certainly need a robots.txt file. We recommend to always use a robots.txt with a default policy.

Howto create a robots.txt file

Select your default policy, want to keep things easy take as default policy allow. Want to customize all URLs a search engine bot is allowed to index choose as default policy disallow.
Select the spiders which this robots.txt applies to. In most circumstances all spiders will do.
Crawl delay means you want to set an interval between multiple crawls on your website per spider.

If you have a sitemap which we recommend you can set the full URL here. Our SEO control panel can create a sitemap for you automatically. Sign-up now to start creating your sitemap.

If you want custom rules for custom spiders enter these rules below custom rules. Your can set rules per spider or per URL.

Robots txt generator