Skip to main content

Robots.txt Builder

Visually configure crawler rules and generate a valid robots.txt file.

How to Use

  1. 1Choose a preset or create rule groups for each User-agent.
  2. 2Add Allow and Disallow paths, one per line or comma-separated; use the path tester to inspect the longest matching rule.
  3. 3Add an absolute sitemap URL, review duplicate, conflict, and block-all warnings, then copy or download robots.txt.

About this tool

User-agent selects a crawler group. Disallow requests that matching paths not be crawled; Allow can reopen a more specific path. Sitemap points crawlers to an absolute sitemap URL. Common mistakes include an empty User-agent, duplicate or conflicting paths, and treating robots.txt as authentication or guaranteed de-indexing. Generation and testing stay in this browser.

Frequently Asked Questions