BrowserToolsFreeExplore all tools
HomeSEO & websiteRobots.txt Generator
SEO & website

Robots.txt Generator

Generate crawler rules and test a URL before publishing.

Let’s get to work

Fill in the details below to get started.

BROWSER TOOL
Use * for all crawlers, or a crawler name such as Googlebot.
Paths start with /. Use * as a wildcard and $ at the end for an exact ending.
Up to 50 paths total. Use %20 for spaces and %23 for #.

About Robots.txt Generator

Create one basic robots.txt group and an optional sitemap. After generating the file, test a URL or path for a chosen crawler. Rules are case-sensitive and apply to the protocol, host and port where the file is published.

How to use it

  1. Enter a user agent and paths to allow or disallow.
  2. Optionally add an absolute sitemap URL and generate the file.
  3. Test a URL against the generated rules, then review and publish the file at the site root.

Frequently asked questions

Does the tester check my live website?

No. It checks the generated text in your browser, not the live robots.txt file or search indexing. Crawlers can interpret edge cases differently.

Does robots.txt keep a page private?

No. It is a crawler instruction, not access control. Protect private content with authentication.

Can a blocked URL still appear in search?

It can appear without its content in some circumstances. Use appropriate index controls on accessible pages when needed.