Robots.txt Generator

Robots.txt at a Glance

How This Tool Works

This generator builds a standard robots.txt file based on your selections. It creates User-agent directives for each bot category and adds Disallow rules for the paths you want to block. The output follows the Robots Exclusion Protocol, which all major search engines and well-behaved crawlers respect.

For Shopify stores, the generated file is designed to be placed in your theme’s robots.txt.liquid template (found under Templates in the Shopify code editor). Shopify serves this file at yourdomain.com/robots.txt. The Shopify-specific defaults block admin, cart, checkout, search, and account pages, which are the same paths Shopify blocks by default.

AI bot blocking works by creating specific User-agent rules for each AI crawler. When GPTBot, ClaudeBot, or other AI training bots encounter a Disallow directive, they are instructed not to crawl your content for training purposes. Note that blocking AI bots does not affect your search engine rankings, as these are separate crawlers from Googlebot and Bingbot.

Step-by-Step Guide: Setting Up robots.txt on Shopify

Configuring a custom robots.txt on Shopify requires a few specific steps that differ from traditional web hosting. Follow this guide to get your file live and verified.

1. Generate your robots.txt using this tool. Select the crawlers you want to allow, check the Shopify paths to block, enable AI bot blocking if desired, and paste your sitemap URL. Click “Generate robots.txt” and review the output.
2. Copy the generated output. Click the “Copy to Clipboard” button to copy the entire robots.txt content. Double-check that your sitemap URL is correct and all desired rules are included.
3. Open the Shopify theme editor. In your Shopify admin, navigate to Online Store, then Themes. Click the three-dot menu on your active theme and select “Edit code.” This opens the theme code editor.
4. Create or edit robots.txt.liquid. In the Templates section, look for robots.txt.liquid. If it does not exist, click “Add a new template” and select “robots.txt” from the template type dropdown. This creates the robots.txt.liquid file.
5. Paste your generated content. Replace the entire contents of robots.txt.liquid with the output from this tool. Save the file.
6. Verify the file is live. Visit yourdomain.com/robots.txt in your browser. You should see your custom rules. The change takes effect immediately.
7. Validate in Google Search Console. Go to Google Search Console, navigate to Settings, then Crawl Stats, then check the robots.txt report. Use the robots.txt Tester to confirm that important pages are not accidentally blocked.
8. Monitor crawl behavior. Over the following weeks, check Google Search Console’s Coverage report to confirm that your desired pages remain indexed and blocked pages are excluded as expected.

Why This Matters for Your Shopify Store

A well-configured robots.txt prevents search engines from wasting their crawl budget on pages that should not be indexed, like your cart, checkout, and internal search results. When Googlebot spends less time on irrelevant pages, it can discover and index your product and collection pages faster, which directly improves your organic visibility.

Blocking AI training bots is increasingly important for content-heavy Shopify stores. If you invest in writing detailed product descriptions, blog posts, or buying guides, AI crawlers can scrape that content and use it to train models that may then generate similar content for your competitors. Blocking these bots protects your intellectual property while keeping your store fully accessible to customers and search engines.

Crawl budget optimization is not just theoretical. Google has confirmed that for large sites (over 10,000 pages), crawl budget becomes a real factor in how quickly new pages are discovered and indexed. Many Shopify stores generate thousands of URL variations through filtered collections, paginated search results, and tag-based URLs. Blocking these low-value URLs ensures Google focuses on your canonical product and collection pages, which are the ones that actually drive revenue from organic search.

Real-World Examples

Here are practical robots.txt configurations for different types of Shopify stores, showing how the rules change based on business needs.

Example 1: Small Apparel Store (Under 500 Products)

A small clothing brand with a blog and 300 products needs basic crawl control. They want full search engine access, AI bot blocking to protect their styled product descriptions, and standard Shopify path blocking.

The custom rule blocking sorted collection views prevents search engines from crawling duplicate versions of collection pages with different sort orders, which can multiply the number of crawled URLs by 5-10x.

Example 2: Large Electronics Store (5,000+ Products)

A high-volume electronics retailer with extensive product filtering needs aggressive crawl budget management. They generate thousands of filtered URLs through price ranges, brand filters, and specification parameters.

This configuration blocks tag-combined collection URLs (the + and %2B patterns), sorted collections, and variant-specific URLs. For a store with 5,000 products across 200 collections, this can reduce crawlable URLs from over 100,000 to under 15,000, dramatically improving crawl efficiency.

Example 3: Content-Heavy DTC Brand with Blog

A direct-to-consumer beauty brand publishes weekly blog content, buying guides, and ingredient education pages. Their content is their competitive advantage, and they need maximum AI protection while keeping search engines fully engaged.

Blocking paginated blog and collection URLs prevents search engines from crawling page 2, 3, 4, etc. of these listings, which contain duplicate links to the same products and articles. The first page of each collection and blog category is sufficient for discovery, with the sitemap handling the rest.

Robots.txt vs Other Crawl Control Methods

Understanding when to use robots.txt versus other methods is essential for effective crawl management. Each approach has distinct strengths and limitations.

For most Shopify stores, the optimal strategy combines robots.txt (to block entire path patterns like /search and /cart), canonical tags (to handle collection filtering and pagination), and sitemap inclusion (to ensure all important pages are discovered). Using all three methods together gives you comprehensive crawl control without relying on any single mechanism.

Common Mistakes to Avoid

• Blocking your sitemap URL in robots.txt. If you add a Disallow rule that accidentally covers your sitemap path, search engines cannot access it even though it is referenced in the same file. Always verify that /sitemap.xml is not blocked by any of your rules. Use Google Search Console’s robots.txt Tester to check.
• Blocking CSS and JavaScript files. Some merchants block /assets/ or other resource directories thinking it will hide their theme. This prevents Googlebot from rendering your pages correctly, causing them to appear as blank pages in Google’s index. Google has explicitly stated that blocking CSS and JS files can negatively impact rankings.
• Using robots.txt to hide sensitive information. Robots.txt is a publicly accessible file. Anyone can visit yourdomain.com/robots.txt and see exactly which paths you are blocking. If you list /pages/wholesale-pricing in your Disallow rules, you are advertising that the page exists. Use password protection for truly sensitive content.
• Forgetting to add the sitemap directive. The Sitemap line in robots.txt is one of the primary ways search engines discover your sitemap. While you can also submit it through Google Search Console and Bing Webmaster Tools, including it in robots.txt ensures every compliant crawler finds it automatically.
• Setting overly broad Disallow rules. A rule like “Disallow: /collections” blocks ALL collection pages, not just filtered versions. Be specific with your rules: use “Disallow: /collections/*?sort_by=” to target only sorted views while keeping the main collection pages crawlable.
• Not updating robots.txt after store changes. When you install a new app that creates public pages, add a new page template, or restructure your navigation, your robots.txt may need updating. A quarterly review ensures new content is properly handled.
• Assuming robots.txt blocks indexing. A common misconception is that blocking a URL in robots.txt removes it from Google. If other sites link to a blocked page, Google may still index the URL (showing a title and description inferred from external links) even though it cannot crawl the actual content. Use meta noindex together with robots.txt for complete removal.

When to Use This Tool

Different situations call for different robots.txt configurations. Use this reference to determine when and how to update your file.

Tips and Best Practices

• Always include your sitemap URL. Adding a Sitemap directive helps search engines discover all your pages efficiently. Your Shopify sitemap is automatically generated at yourdomain.com/sitemap.xml and includes products, collections, pages, and blog posts.
• Do not block CSS or JavaScript files. Googlebot needs to render your pages to understand their content. Blocking access to CSS or JS files can prevent Google from seeing your pages correctly, which hurts your rankings.
• Test your robots.txt before deploying. Use Google Search Console’s robots.txt Tester to verify that your rules do not accidentally block important pages. A misplaced Disallow rule can de-index your entire product catalog.
• Remember that robots.txt is advisory, not enforcement. Well-behaved bots honor robots.txt, but malicious scrapers ignore it. For truly sensitive content, use password protection or Shopify’s built-in access controls instead.
• Review your robots.txt quarterly. As you add new pages, sections, or apps to your store, revisit your robots.txt to ensure new content is accessible to search engines and unwanted paths remain blocked.
• Use specific path patterns rather than broad blocks. Instead of blocking an entire directory, target specific URL parameters or patterns. This gives you precise control without accidentally blocking important content.
• Keep your robots.txt file organized with comments. Adding comment lines (starting with #) to explain each section makes it easier for you or your developer to understand and maintain the file over time.

Related Tools

• robots.txt AI Bot Checker – Analyze any website’s robots.txt to see which AI bots are currently blocked or allowed.
• Sitemap Checker – Validate your Shopify sitemap to ensure all important pages are included and properly formatted.
• Shopify SEO Checker – Run a comprehensive SEO audit on your Shopify store, including robots.txt validation, meta tags, and page speed.