SEO & marketing
Works Offline — No Internet Required
Robots.txt Generator — Crawler Rules, Sitemaps & AI Bots
Generate robots.txt with Allow/Disallow groups, sitemap URLs, Next.js and WordPress presets, and AI crawler blocks. Validation stays in your browser.
- Mobile Friendly
- Accessibility Tested
- SEO Optimized
- Fast Loading
- Responsive
Version 1.0.0 · Last reviewed
How it worksShow guide
Introduction
robots.txt is a public crawl policy at the site root. This builder writes User-agent groups, Allow/Disallow paths, optional Crawl-delay, and Sitemap lines. It also flags Google’s Crawl-delay ignore and invalid sitemap URLs. The file is generated on your device.
Step-by-step
- Start from a preset or an empty group
- Add user-agents, Allow/Disallow paths, and sitemap URLs
- Copy or download robots.txt after reviewing warnings
Worked example
The Next.js preset allows / and disallows /api/, /_next/, and /preview/, then appends a sitemap URL you can replace with your domain.
Use cases
- Ship a first robots.txt for a new static site
- Block wp-admin or /_next/ without blocking public pages
- Disallow known AI training crawlers while keeping search bots
Frequently asked questions
Does robots.txt hide a page from Google?
No. Disallow means “please do not crawl.” A URL can still appear in search if it is linked elsewhere. Use noindex (and authentication) when you need a page unpublished.
Do AI crawlers honor robots.txt?
Many publish a user-agent and say they honor robots.txt; that is a policy, not a guarantee. Review each vendor’s current documentation.
Where do I put the file?
Serve it at https://your-domain/robots.txt (site root). Subfolder robots.txt files are not the standard location search engines look for.
Is Crawl-delay supported?
Some crawlers honor it. Google does not. This tool warns when you set Crawl-delay on Googlebot.
Related tools
-
JSON-LD Schema Markup Generator
Build schema.org JSON-LD for FAQPage, Organization, Article, BreadcrumbList, SoftwareApplication, and Person — privately in your browser.
-
Meta Tag Generator
Generate title, description, Open Graph, Twitter, and canonical tags from your page details.
-
SEO Search Engine Preview
Preview estimated search snippets across major search engines and validate your SEO metadata.
-
UTM Builder
Build campaign URLs with utm_source, utm_medium, utm_campaign, and optional term/content.
Popular tools
Was this helpful?
Your feedback stays on this device only — it is not sent to the Bestrao team yet.