Edit robots.txt

Robots.txt tells search engine crawlers which URLs they may crawl.

Where you can find your site's robots.txt file

Visit /robots.txt on your domain to see the rules your site serves. The response can come from a physical file in your site's webroot (root directory) or be generated dynamically by WordPress when no file exists.

SEO plugins such as Rank Math and Yoast SEO can modify WordPress's generated response. Rank Math can also replace it with custom rules saved in the WordPress database. Deleting a physical file does not remove those saved rules.

How to edit the content of robots.txt

If you use an SEO plugin to manage robots.txt, check its saved rules first. You do not need a physical file to customize the dynamically generated response.

To edit a physical file, follow our guide on file access, navigate to your site's webroot, and edit robots.txt. You can also upload a physical file there to replace the dynamically generated response.

After changing the rules, purge Templ Cache and check /robots.txt on your domain again.

Troubleshoot unexpected rules

These rules block crawling of the entire site for crawlers following this group. They do not protect private content or reliably remove pages already listed in search results:

User-agent: *
Disallow: /

If you see unexpected rules:

  1. Check for a physical robots.txt in your site's webroot using File Manager or SFTP. If it exists, inspect its contents.
  2. In WordPress, check Settings → Reading → Search engine visibility. For a public site you want indexed, Discourage search engines from indexing this site should be unchecked.
  3. Check your SEO plugin's saved robots.txt rules. Rank Math can replace the generated response with saved rules that block crawling even when WordPress's search engine visibility setting is unchecked.
  4. Compare the normal response with a request that bypasses Templ Cache (see below). If the unwanted rules remain with X-Cache-Status: BYPASS, Templ Cache is not serving an old copy for that request.
  5. Correct the physical file or saved plugin rules, purge Templ Cache, and verify the ordinary /robots.txt URL again.

If Rank Math's saved custom rules are the cause, clear them to return to the dynamically generated response, or replace them with your intended rules. Preserve any custom rules you still need. Other plugins may still modify the generated response.

Check the response without Templ Cache

Run this from a terminal, replacing your-domain.com with your domain:

curl -i 'https://your-domain.com/robots.txt?nocache=123456'

Check for X-Cache-Status: BYPASS in the response headers. Incognito mode does not bypass server-side caching, and a confirmed Templ Cache bypass does not rule out independently configured third-party caches, such as Cloudflare.

After correcting the rules and purging Templ Cache, verify the ordinary /robots.txt URL without the query string.

FAQ

Why does robots.txt still exist after I delete the file?

WordPress generates /robots.txt dynamically when no physical file exists. An SEO plugin can modify or replace that response using settings saved in the database, so deleting the file does not necessarily remove the unwanted rules.

Why does Rank Math report a physical file when I cannot find one?

An editor warning alone does not confirm that a physical file exists. If File Manager and the plugin disagree, contact support.