Robots.txt Generator

Build and validate robots.txt directives.

Privacy: Drafts and generated markup stay in your browser unless you explicitly save them to your account.

Crawler directives

One disallow path per line. Review rules before deployment because robots.txt is public.

User agent

Allow path

Sitemap URL

Disallow paths

robots.txt

User-agent: *
Allow: /

Support

What this tool does

Draft crawler directives for a robots.txt file.

How to use it

Choose a user agent, add allow/disallow paths and an optional absolute sitemap URL, then inspect the text.

Worked example

Disallow /private/ produces a Disallow: /private/ directive for the selected agent.

Limitations and checks

Robots rules are not authentication and do not reliably remove indexed URLs. Do not block resources required to render public content.

How your data is processed

The operation runs in your browser. Copy and download actions use your device. Where Save work is offered, a local copy is stored; eligible work can sync only under your enabled account settings. See Privacy for site analytics and storage details.

Privacy and storage details · Report an incorrect result · Browse tools