Generate Your Robots.txt File
Rule 1
No disallow paths (all pages allowed)
The Ultimate Free Robots.txt Generator
Master your website's crawl budget and SEO visibility with our advanced Robots.txt Generator. Whether you are running a small brochure site or a massive e-commerce store, this tool gives you precise control over how Google, Bing, and other bots interact with your site.
Table of Contents
What Is a robots.txt File?
A robots.txt file is a simple text file placed in the root directory of your website (e.g., https://example.com/robots.txt). It acts as a gatekeeper, giving instructions to search engine bots (crawlers) about which pages they are allowed to visit and which they should ignore.
It is the first file a legitimate bot requests when it visits your site. Think of it as a "Code of Conduct" sign at the entrance of your digital property. While it doesn't physically block access (bad bots can ignore it), reputable search engines like Google, Bing, and Yandex follow its rules strictly.
Why is robots.txt Critical for SEO?
Crawl Budget Optimization
Search engines have a limited "budget" for how many pages they crawl on your site per day. If they waste time crawling low-value admin pages or tags, they might miss your important pages. Robots.txt prevents this waste.
Private Content Protection
Keep staging environments, admin dashboards, and checkout pages out of search results. While not a security mechanism, it keeps these pages from cluttering public search indexes.
Preventing Duplicate Content
If your site generates print-friendly versions or dynamic URLs with parameters, robots.txt can tell Google to ignore them, preventing "duplicate content" penalties.
Sitemap Discovery
It's the standard place to link your XML sitemap. This gives crawlers a direct map to all your detailed content, ensuring faster indexing of new pages.
How to Use Our Generator
- 1
Set Default Permissions
Start with the "All Robots" section. By default, you usually want to "Allow" all robots. Only switch this to "Disallow" if you want your entire site to remain hidden (e.g., if it's under construction).
- 2
Exclude Specific Bots (Optional)
If you specifically want to block marketing bots (like MJ12bot or AhrefsBot) to save server resources, select them from our list and choose "Refuse".
- 3
Add Restricted Directories
In the "Restricted Directories" section, type in the paths you want to hide. Common examples include:
- /wp-admin/
- /checkout/
- /tmp/
- /private/
- 4
Link Your Sitemap
Paste the full URL of your sitemap (e.g.,
https://yoursite.com/sitemap.xml). This is a best practice that helps Google discover your pages faster.
Understanding the Syntax
User-agent
Specifies which bot the following rules apply to. User-agent: * means "all bots". User-agent: Googlebot applies only to Google.
Disallow
Tells the bot NOT to visit this path. Disallow: /images/ prevents crawling of your images folder.
Allow
Used to override a Disallow rule. For example, you might Disallow /wp-admin/ but Allow /wp-admin/admin-ajax.php.
Sitemap
Specifies the location of your XML sitemap. This directive is supported by all major search engines.
Deadly Mistakes to Avoid
- Accidentally Blocking the Whole Site
Typing
Disallow: /(with the slash) tells bots to ignore your ENTIRE website. Only use this if you truly want to disappear from Google. - Blocking CSS and JS Files
Google needs to "render" your page to understand if it's mobile-friendly. If you block your
/assets/or/css/folders, Google sees a broken page and might rank you lower. - Using Robots.txt for Security
Never use this file to hide private data. Hackers check robots.txt specifically to see what you are trying to hide. Use password protection (.htaccess) for real security.
Frequently Asked Questions
Can I have multiple robots.txt files?
No. You must have exactly one file named robots.txt (all lowercase) in the root directory. Subdomains can and should have their own robots.txt file.
How long does it take for changes to take effect?
Google bots usually cache your robots.txt file for about 24 hours. You can force a refresh using the "Robots.txt Tester" tool in Google Search Console if you need immediate updates.
Does "Disallow" prevent indexing?
Not always. "Disallow" prevents crawling (visiting the page), but if a lot of other sites link to that page, Google might still index it and show the URL in search results (usually without a description). To prevent indexing entirely, use the noindex meta tag on the page itself.
What is Crawl-delay?
Some bots (like Bingbot) support a Crawl-delay: 10 directive, which tells them to wait 10 seconds between requests. This helps prevent your server from crashing if too many bots visit at once. Googlebot ignores this directive (you configure Google's crawl rate in Search Console instead).
Example robots.txt Files
Seeing real examples makes the syntax click. Here are a few common setups you can adapt to your own site.
Allow Everything
The most common setup, letting all bots crawl the whole site and pointing them to your sitemap.
User-agent: * Disallow: Sitemap: https://yoursite.com/sitemap.xml
Block an Admin Area
Crawl the site but keep a private folder out of search results.
User-agent: * Disallow: /admin/ Disallow: /checkout/ Sitemap: https://yoursite.com/sitemap.xml
Block the Entire Site
Used only for staging or sites under construction. Remove this before going live.
User-agent: * Disallow: /
robots.txt vs the Noindex Meta Tag
One of the most common points of confusion is the difference between blocking a page in robots.txt and stopping it from being indexed. They sound similar but do different jobs, and mixing them up can produce surprising results.
A Disallow rule in robots.txt stops a bot from crawling a page, meaning it will not read the page content. However, if other sites link to that page, Google can still list the URL in search results, often without a description. So robots.txt controls crawling, not indexing.
The noindex meta tag, placed in the HTML of the page itself, is what reliably keeps a page out of the index. The catch is that Google must be able to crawl the page to see the noindex tag. If you block the page in robots.txt, Google never reads the tag, so the two methods can work against each other.
The rule of thumb: use robots.txt to manage crawl budget and keep bots away from unimportant areas, and use the noindex tag when you specifically need a page kept out of search results.
How to Test Your robots.txt
A small mistake in this file can have an outsized effect, so testing before and after you publish is worth the few minutes it takes.
View it in a browser. Open
yoursite.com/robots.txtdirectly. It should display the rules as plain text exactly as you wrote them.Use Search Console. Google's robots.txt report shows how Google reads your file and flags errors or warnings.
Check key URLs. Confirm that important pages are allowed and that only the areas you meant to block are disallowed.
Verify CSS and JS load. Make sure you have not blocked the asset folders Google needs to render and understand your pages.
Platform-Specific Tips
WordPress
WordPress generates a virtual robots.txt by default. Most SEO plugins let you edit it without touching files. A common setup blocks /wp-admin/ while allowing admin-ajax.php, and links your sitemap.
Shopify
Shopify provides a sensible default robots.txt and now allows edits through a theme template. Be cautious, since the defaults are already tuned for most stores.
Static Sites
For hand-built or framework sites, place the robots.txt file in your public or root directory so it is served at the top level of your domain.
Wix and Squarespace
These platforms manage robots.txt for you and offer limited controls in their SEO settings. Generating a reference file helps you understand what they produce.