What Does robots.txt Do for a WordPress Site?

How robots.txt Actually Works

When a compliant crawler visits a site, it checks robots.txt first and respects any Disallow rules found there, skipping those paths during its crawl. WordPress generates a basic version automatically, and most SEO plugins let you customize it further — commonly excluding admin areas, certain query-string variations, or duplicate-content-prone URLs from being crawled unnecessarily.

Common Misunderstandings

• It's not a security mechanism — a malicious actor or a non-compliant scraper can simply ignore robots.txt entirely, and disallowed URLs remain fully accessible to anyone who has or guesses the direct link.
• Disallowing a URL in robots.txt doesn't guarantee it won't be indexed — if other sites link to that URL, search engines can still index it (typically showing it without a description, since they couldn't crawl the actual content), which is a common source of confusion.
• The correct way to prevent indexing of a specific page is a noindex meta tag on that page itself, not a robots.txt disallow rule — these solve different problems and are sometimes confused for each other.
• Overly aggressive robots.txt rules can accidentally block search engines from crawling content that should be indexed, which is worth checking periodically rather than assuming the default configuration is always correct for the site's specific needs.

Using robots.txt Correctly

1. Use robots.txt to manage crawl efficiency (excluding low-value URL patterns), not as a security or access-control mechanism.
2. Use a proper noindex tag for pages that genuinely shouldn't appear in search results.
3. Periodically review the file to confirm it isn't accidentally blocking content that should be crawled.

Need your robots.txt and indexing configuration reviewed properly? See WordPress bug fix.