Robots.txt Tester & Validator

Robots.txt is a text file that tells search engine crawlers which pages or sections of your website they should not access. It helps prevent crawlers from indexing duplicate content, admin pages, or pages that consume server resources. Every website should have a robots.txt file, even if it just allows all access.

The robots.txt file must be placed at the root of your domain: https://example.com/robots.txt. It will not work in subdirectories. Each subdomain requires its own robots.txt file (e.g., https://blog.example.com/robots.txt).

No. Robots.txt prevents crawlers from accessing pages, but does not guarantee they will not appear in search results. If other sites link to a blocked page, search engines may still index the URL without crawling the content. To prevent indexing, use the noindex meta tag or X-Robots-Tag HTTP header.

Disallow blocks crawler access to a path. Allow explicitly permits access, which is useful for creating exceptions to broader Disallow rules. For example, you can block /admin/ but allow /admin/public/.

Yes. Google and most modern crawlers support * (matches any sequence of characters) and $ (denotes end of URL). For example, Disallow: /*.pdf$ blocks all PDF files. Wildcard support is not universal, so test your rules carefully.

Use this tool to validate syntax and test paths. Google Search Console also provides a robots.txt tester under the "Crawl" section. You can also view your live robots.txt file by visiting https://yoursite.com/robots.txt in a browser.

Crawlers use the most specific matching user-agent section. If you have rules for User-agent: Googlebot and User-agent: *, Googlebot will follow its specific rules and ignore the * section. Other bots will use the * rules.

No. Google recommends allowing access to CSS and JavaScript files so crawlers can properly render and understand your pages. Blocking these files may negatively impact how search engines interpret your site.

No. Robots.txt is a voluntary standard. Malicious bots ignore it. For security, use proper authentication, rate limiting, CAPTCHA, and server-level access controls. Robots.txt is only for legitimate crawlers that respect the standard.

Crawl-delay specifies the minimum number of seconds between requests from a crawler. Bing and some other search engines support it, but Google does not. Google uses its own algorithms to determine crawl rate, which you can configure in Google Search Console.

Create separate User-agent sections. For example, to block BadBot but allow all others:

User-agent: BadBot
Disallow: /

User-agent: *
Disallow:

Not directly, but improper use can harm SEO. Blocking important pages prevents them from being indexed. Blocking duplicate or low-quality content can help search engines focus on your best pages. Always test changes carefully.