Shopify does not offer editing of robots txt and maintains the files itself. This is troublesome for SEO but you can hide the old product pages by just adding the ‘noindex’ tag into the required page.

Sadly, as of 2021, we do not see this issue of editing robots.txt being resolved ever since it was raised. However, in order to serve Google better, Shopify store owners have found ways to address this issue. 

In this blog post, we will share what is robots.txt and how to hide old product pages on Shopify robots.txt file and also how to verify if the robots.txt is updated. 

What is robots.txt and how it could affect my store?

Google uses robots or we can say ‘crawlers’ to find your store product pages. When your pages are crawled by the Google robots, it helps in ranking on Google. Simply put, if Google can’t find your pages, it won’t be able to show it search results. 

Every store has its robots.txt file which tells Google crawlers which pages should be crawled and which shouldn’t. However, this doesn’t mean robots.txt file keeps your pages ‘not to show’ but it avoids overloading of your website/store. 

Probably, your store has too many pages indexed even if you don’t think you do, must check. You will get surprised. Point is, if a search engine crawls any store, it would crawl every single page.

If your store has a lot of pages, the search engine bot/crawler will consume more time which can negatively affect your store rankings overall. 

Do you know Googlebot has a crawl budget?

Crawl budget is the number of URLs a bot crawls. This budget depends on two factors. First the crawl rate limit, here’s how Google explains it:

Shopify robots.txt

Second is crawl demand and here’s what it means:

Shopify robots.txt

In our use case where we don’t want old or outdated pages to show in Google results, we can use the ‘noindex’ tag in any of the required product pages. 

What is a noindex tag?

A tag which tells Google not to crawler any specific product page and usually embedded in code like this:

You may be thinking where to add this code when Shopify doesn’t allow it. Hold your horses, there is a hope. First, you need to locate your robots.txt file in the Shopify store. 

Where can I find robots.txt file?

Find your store’s robots.txt file in the root directory of your store’s primary domain name. 

Example: 

phaedrasolutions.com/robots.txt

List of pages must be blocked by robot.txt

prettythings.com/checkout

prettythings.com/checkouts

prettythings.com/account/register

prettythings.com/account/login

prettythings.com/finishorder

prettythings.com/postreview

prettythings.com/productimage

prettythings.com/productupdates

prettythings.com/thankyou

 Why is Google banning indexing my product page URLs?

The obvious reason is to prevent duplicate pages from indexing.

Oftentimes you misread the robots.txt file. There could be several instances when Shopify won’t allow inappropriate URLs. 

There could be many product pages which are duplicate – just think that you have 10 tags. There are 1,025 combinations of those 10 tags. And most of those combinations are nonsensical or don’t add a lot of value — that’s why Shopify is preventing them from being indexed.

‘Noindex technique is recommended by Google’ for hiding/removing old pages

One of the best ways to hide your product pages from being indexed is using a noindex tag as we were referring to before. 

1-Go to your admin dashboard and enter into themes section. Select ‘edit code’.

Shopify robots.txt

2-Choose the theme you want to edit which is theme.liquid.

Shopify robots.txt

3-Enter the following code in the head section. We used /baby-clothes’ in the category name as shown in the screenshot. 

{% if product.url == ‘/products/categoryname’ %}

<meta name=”robots” content=”noindex”>

{% endif %}

Shopify robots.txt

Hit save on the top right corner of the screen.

Verify if the robots.txt is updated by using the tester tool

Finally test your robots.txt file in order to be sure that old or outdated pages are not ranking anymore. Google provides a free tool i.e. robots.txt tester as part of the Webmaster tools. 

Add the updated robots.txt there and test. Remember, changes in actual code may take a day or so. 

Final verdict

SEO of a Shopify store is often tough given the fact that you can’t edit the robots.txt file directly. You have to look into hacks in order to do so. However, editing Shopify robots.txt may be a daunting task if you have to hide multiple product pages. Just make a list of all the stale/duplicate product pages and convert the page names into page handles. Adding {% else if handle contains ‘new-page-handle’ %} in the head section may not be daunting if there are only a few product pages you wish to hide. 

In case you are overwhelmed with the duplicate product pages and can’t handle the robots.txt file yourself, we are always happy to help. 

Get In Touch With Our Experts