Robots.txt Generator
Draft a general robots.txt policy with path and sitemap checks, broad-block warnings, and copyable output.
Default policy
Google does not support this directive. Check support for other crawlers separately.
The URL is not fetched, and its domain is not compared with your site.
Place the file at your site root as robots.txt (example.com/robots.txt). It controls crawling, not access: protect private data with authentication. A blocked crawler cannot read a page-level noindex directive. Inputs stay in your browser; the sitemap address is not requested.
Example: allow a resource within a restricted folder
With the allow-by-default policy, enter /example-folder/ under Disallow and /example-folder/public.css under Allow. The output adds an exception for that resource. These are examples; choose and test your own URL paths before copying them.
Pre-release checks
This tool does not test a live site's rules or verify that Google fetched the file. After applying it at the site root, check the HTTP response, intended crawler group, and representative important blog, service, and resource URLs. Keep the previous file and a rollback plan.
User-agent: *
Disallow: /example-folder/
Allow: /example-folder/public.css
Sitemap: https://example.com/sitemap.xmlAre crawl and indexing signals configured for your site?
Review robots.txt, sitemap, and indexing signals together with a technical SEO audit.
TOOL GUIDE
What does this robots.txt generator check?
Build a draft for one general crawler group (User-agent: *). The tool validates URL paths and a sitemap URL, flags broad blocks and path conflicts, and produces plain-text rules locally in your browser.
This is a text generator, not a robots parser, crawler, or live-site test. It does not request your sitemap or verify the robots.txt file served by a site. A robots.txt rule manages crawling, not access to private data or guaranteed removal from search results.
How to use it
- Choose allow-by-default or block-all. The latter emits Disallow: / and disables the separate path fields.
- Enter one root-relative URL path per line. Add an optional absolute HTTP or HTTPS sitemap URL; it is validated as text but never requested.
- Read every warning and compare the rules with important real URLs before applying the output at the site's top-level /robots.txt path. Keep a copy of the previous file for rollback.
When to use it
- Draft a general crawler policy for paths that should not be crawled, such as account or cart areas, after confirming the actual URL patterns.
- Allow a specific resource inside a disallowed folder and inspect the exact generated directives.
- Prepare a sitemap directive while keeping the sitemap and robots URL checks separate from a live HTTP test.
This tool processes supplied data in your browser. Text and inputs are analyzed on your device without being sent to a server.
Frequently asked questions
Does robots.txt protect private pages or remove them from Google?
No. Disallow asks compliant crawlers not to fetch matching paths; it is not authentication or access control. A blocked URL can still be indexed without its content. To use a page-level noindex directive, crawlers must be able to fetch and read that directive. Protect confidential data with authentication.
Does Google support Crawl-delay?
Google's robots.txt parser does not support Crawl-delay. Other crawlers may differ; check their current official documentation before relying on the extension.
Does this tool verify that my rules work on the site?
No. It only validates input shape and generates text. After deployment, fetch the top-level robots.txt over HTTP and test important URL examples with an appropriate parser or crawler. This page does not make network requests.