Robots.txt Generator
Easily generate a robots.txt file to control how search engines crawl your website.
Global Settings
Crawler Rules 1
robots.txt
User-agent: *
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xml
About Robots.txt
A robots.txt file gives instructions about your site to web robots; this is called The Robots Exclusion Protocol.
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.
Where should I put my robots.txt file?
The robots.txt file must be placed at the root of your website host to which it applies. For example, to control crawling on all URLs below https://www.example.com/, the robots.txt file must be located at https://www.example.com/robots.txt.
What does "User-agent: *" mean?
The asterisk (*) is a wildcard that applies the rule to all web crawlers. So "User-agent: *" means the instructions that follow apply to every bot that visits your site.
Can I have multiple sitemaps in my robots.txt?
Yes! You can add as many "Sitemap: [URL]" directives as you need anywhere in your robots.txt file. This helps search engines discover all your XML sitemaps efficiently.
Il est temps de laisser tomber Google Analytics.
Vous en avez assez de la frustration, de la complexité et des problèmes de confidentialité de Google Analytics ? Nous aussi. C'est pourquoi nous avons créé Swetrix : l'alternative éthique, open source et entièrement sans cookies.