Web tool · FREE · NO ACCOUNT

WordPress Robots.txt Tester and Generator

Find out which rule allows or blocks a page before editing your robots.txt file. Load a public website or paste the file, choose a crawler, and test a URL. The result shows the matching directive and line number. You can also generate a starter file for WordPress.

Generate a WordPress robots.txt file
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Rules are tested locally. Loading a website sends its URL to our server. Robots.txt controls crawling, not whether a URL appears in search results.

How to use robots.txt tester

  1. Load robots.txt from your public website or paste its contents into the editor.
  2. Enter a full page URL or path, such as /wp-admin/admin-ajax.php, and the crawler product token.
  3. Review the matched rule and any warnings. If you generate a file, add your actual sitemap URL.
  4. Back up your existing file, apply the change, and test the live file again.

Questions and limits

Does robots.txt prevent a page from being indexed?

No. It controls crawling. A blocked URL can still appear in search results. Use a suitable noindex directive on a crawlable page when you want to prevent indexing.

How are conflicting rules resolved?

The tester selects the most specific matching user-agent groups, then the longest matching path rule. Allow wins when equally specific rules conflict. Wildcards and end-of-path markers are supported.

Should I block wp-content?

Avoid blocking assets that crawlers need to render your pages, including theme CSS, JavaScript and images. The starter file blocks wp-admin while allowing admin-ajax.php.

Does the tool edit my WordPress site?

No. Loading a website is read-only. Pasted text and rule tests run in your browser. Generated output must be applied by you.