Explainers

Understand how robots.txt works, what each directive does, and how crawlers interpret your rules.

Robots.txt controls which pages search engines can crawl. These articles explain how each directive works, what crawlers actually do with your rules, and the difference between blocking crawling and blocking indexing.

For a comprehensive overview, see our The Complete Robots.txt Guide.

Do You Need a robots.txt File?

Does your website need a robots.txt file? When it's essential, when it's optional, and what happens if you don't have one.

How robots.txt Affects Your SEO

How robots.txt impacts search engine optimization. Crawl budget, indexing, and the SEO mistakes that robots.txt can cause.

Crawl-Delay in robots.txt Explained

What the Crawl-delay directive does, which search engines support it, and when you should (and shouldn't) use it.

robots.txt Disallow Directive Explained

How the Disallow directive works in robots.txt. Syntax, examples, path matching, and common mistakes that accidentally block your entire site.

Noindex in robots.txt: Why It Doesn't Work

Google no longer supports the noindex directive in robots.txt. What happened, what to use instead, and how to properly deindex pages.

robots.txt User-Agent: How to Target Specific Crawlers

How to use the User-agent directive in robots.txt to create rules for specific search engines, bots, and crawlers.

robots.txt vs Meta Robots Tags: Which to Use

The difference between robots.txt and meta robots tags (noindex, nofollow). When to use each, and why using the wrong one can hurt your SEO.

Wildcards in robots.txt: Using * and $ Patterns

How to use wildcard patterns in robots.txt. The * and $ characters, path matching, and practical examples for complex blocking rules.

What Does robots.txt Actually Do?

What robots.txt does and doesn't do. How crawlers use it, why it's advisory not enforceable, and the limits of robots.txt.

What Is robots.txt?

What robots.txt is, how it works, and why every website should have one. The complete introduction to the robots exclusion protocol.