How to Edit Robots.txt on Shopify: A Practical Guide

Robots.txt tells search engine crawlers which parts of your site they should and should not access. Shopify provides a default robots.txt that handles the basics, but customizing it can improve crawl efficiency—especially for larger stores.

What Is Robots.txt?

The robots.txt file lives at `yourwebsite.com/robots.txt`. Crawlers read it before crawling your site to understand which URLs they are allowed to access. It uses simple directives:

• `User-agent:` — Which crawler the rules apply to
• `Disallow:` — URLs the crawler should not access
• `Allow:` — Exceptions to disallow rules
• `Sitemap:` — Location of your XML sitemap

How to Edit Robots.txt on Shopify

1. Go to Online Store > Themes.
2. Click Actions > Edit Code (or the equivalent in your theme editor).
3. Click Add a new template and select robots.txt.
4. This creates a `robots.txt.liquid` file you can customize.

Understanding the Default Template

The default `robots.txt.liquid` iterates over Shopify's built-in rules using `robots.default_groups`. Your customizations go inside this loop.

What to Add

Block Low-Value URLs

For the `*` (all bots) user agent, common additions:

```liquid {%- if group.user_agent.value == '*' -%} {{ 'Disallow: /collections/all*' }} {{ 'Disallow: /*?q=*' }} {{ 'Disallow: /collections/*/*' }} {{ 'Disallow: /blogs/*/tagged/*' }} {%- endif -%} ```

• `/collections/all*` — The "all products" page; thin and unhelpful.
• `/*?q=*` — Vendor and type query pages (`/collections/vendors?q=Name`).
• `/collections/*/*` — Product tag filter pages. Only add this after fixing internal product links to use `/products/handle` paths.
• `/blogs/*/tagged/*` — Blog tag pages.

Remove a Default Rule

If you want to allow crawling of a page Shopify blocks by default (e.g., `/policies/`):

```liquid {%- for rule in group.rules -%} {%- unless rule.directive == 'Disallow' and rule.value == '/policies/' -%} {{ rule }} {%- endunless -%} {%- endfor -%} ```

Block Specific Bots

Add custom user-agent blocks at the end of your file:

``` User-agent: ia_archiver Disallow: / ```

Add Extra Sitemaps

If you have a custom image sitemap or an additional sitemap from a migration:

``` Sitemap: https://yourwebsite.com/custom-sitemap.xml ```

Why Customize Robots.txt?

Crawl Budget

Search engines allocate limited crawl resources to each site. If Googlebot spends time crawling thousands of thin tag pages, it has less capacity for your actual products and collections. Blocking low-value pages redirects crawl budget to pages that matter.

Thin Content Prevention

Even if you use canonical tags or noindex on thin pages, crawlers still visit them. Robots.txt prevents the visit entirely, which is the most efficient approach for crawl budget.

Caution

Robots.txt is powerful and mistakes can be costly:

• Do not block URLs you want indexed. `Disallow: /collections/*/*` blocks product URLs accessed through collections. This is safe only after you have fixed internal links.
• Robots.txt does not remove pages from the index. If a page is already indexed, blocking it in robots.txt prevents de-indexing. Use noindex for pages that need removal from the index.
• Test changes carefully. Use Google Search Console's robots.txt tester to verify your rules before deploying.

Full Template Example

```liquid {% for group in robots.default_groups %} {{- group.user_agent -}} {% for rule in group.rules %} {{- rule -}} {% endfor %} {%- if group.user_agent.value == '*' -%} {{ 'Disallow: /collections/all*' }} {{ 'Disallow: /*?q=*' }} {{ 'Disallow: /collections/*/*' }} {{ 'Disallow: /blogs/*/tagged/*' }} {%- endif -%} {%- if group.sitemap != blank -%} {{ group.sitemap }} {%- endif -%} {% endfor %} ```

Next Steps

1. Fix internal product links to use `/products/handle`.
2. Create the `robots.txt.liquid` template.
3. Add the blocking rules above.
4. Test in Google Search Console's robots.txt tester.
5. Monitor crawl stats in GSC over the following weeks.