Reference
Syntax references, directive glossaries, example files, and bot user-agent lists.
Quick-reference material for robots.txt syntax, directives, and crawler user-agent strings. Useful when you know what you're looking for and need the exact format or value.
robots.txt Glossary: Every Directive and Term Explained
Definitions of every robots.txt directive and related term. User-agent, Disallow, Allow, Sitemap, Crawl-delay, and more.
Read morerobots.txt Examples for Every Situation
Copy-paste robots.txt examples: block all bots, allow all bots, block specific directories, WordPress, Shopify, e-commerce, and more.
Read morerobots.txt Syntax Reference
Complete robots.txt syntax reference. Every directive, pattern, and rule explained with examples. Bookmark this page.
Read more