Is your Shopify robots.txt blocking Google Shopping? (Storebot-Google)

Google's crawler documentation says rules addressed to Storebot-Google "affect all surfaces of Google Shopping", and that its common crawlers "always obey robots.txt rules" (Google: common crawlers). So a robots.txt rule that blocks your product URLs takes them out of Shopping as surely as a 404.

Shopify's default is fine

Shopify generates a default robots.txt it describes as "optimal for SEO": public content such as product, collection and policy pages is allowed, while admin, cart, checkout and account pages are blocked (Shopify help). Problems start with a custom robots.txt.liquid template — added by a theme, an app or an old SEO tweak.

How to check

  1. Open yourstore.com/robots.txt.
  2. Find the User-agent: * group, and any group for Googlebot or Storebot-Google (Google uses the most specific group that names it).
  3. Look for a Disallow covering /products or /. When rules overlap, the longest matching rule wins, and Allow wins a tie.

How to fix

Shopify: delete the custom robots.txt.liquid in the theme code editor to go back to the default; edits to that file are an unsupported customization (source).

Our $29 full audit checks robots.txt for Googlebot and Storebot-Google against every product URL it samples (up to 20), along with those product pages, your policies and contact details.

Check your four Shopify policy pages for free: Run the free policy check

Not affiliated with Google or Shopify.