Today I’m going to show you how to create and manage robots.txt file in Magento 2.
Robots.txt is an all important file from the SEO perspective which contains instructions for the web crawlers i.e. search engines as to which pages to crawl and which not. This one tiny file can get your entire store or specific pages out of search index if configured incorrectly. There will be sections or pages in your store which you won’t want to be indexed and robots.txt file is the place to communicate this to search engines. Almost every website needs this file and so do Magento 2 stores.
An easy way to locate your website robots txt file is to access the file via the following path:
https://domain.com/robots.txt
If you haven’t configured, you’ll either get an empty page or you’ll land on a 404 error page. In this case you need to create & configure the Robots.txt file.In Edit Custom instruction of robots.txt File, enter any custom instructions for the file if you feel the need for it. For example, you might want to disallow access to all folders while you are still developing your store. Refer below for some custom robots.txt instructions.
The Reset To Defaults button will reset the robots.txt file to the default, removing all the custom instructions.
Once you’re done, click Save Configuration button to apply the changes.
You can also use No-Index No-Follow Magento 2 extension to automatically set Meta Robots Tags for product, category & CMS pages.User-agent:* Disallow:
User-agent:* Disallow: /
User-agent: Bingbot Disallow: /foldername/
User-agent: Bingbot Disallow: /foldername/restricted-webpage.html
1. Disallow: /lib/ 2. Disallow: /*.php$ 3. Disallow: /pkginfo/ 4. Disallow: /report/ 5. Disallow: /var/ 6. Disallow: /catalog/ 7. Disallow: /customer/ 8. Disallow: /sendfriend/ 9. Disallow: /review/ 10. Disallow: /*SID=
1. Disallow: /checkout/ 2. Disallow: /onestepcheckout/ 3. Disallow: /customer/ 4. Disallow: /customer/account/ 5. Disallow: /customer/account/login/
1. Disallow: /tag/ 2. Disallow: /review/
1. Disallow: /app/ 2. Disallow: /bin/ 3. Disallow: /dev/ 4. Disallow: /lib/ 5. Disallow: /phpserver/ 6. Disallow: /pub/
1. Disallow: /catalogsearch/ 2. Disallow: /catalog/product_compare/ 3. Disallow: /catalog/category/view/ 4. Disallow: /catalog/product/view/
1. Disallow: /*?dir* 2. Disallow: /*?dir=desc 3. Disallow: /*?dir=asc 4. Disallow: /*?limit=all 5. Disallow: /*?mode*
Now, that you have created the Robots.txt file, you can check & validate the file via Google’s Robots.txt Testing tool. With this tool, you can identify & validate robots.txt file errors.
User-agent: Googlebot User-agent: Googlebot-Image/1.0 User-agent: Googlebot-Video/1.0 User-agent: Bingbot User-agent: DuckDuckBot User-agent: YandexBot User-agent: Baiduspider User-agent: ia_archiver #Alexa User-agent: Slurp #Yahoo
As stated above, robots.txt file instructs web crawlers how to index your website. It is the way to communicate with search engines. To make sure your Magento 2 store pages are indexed and displayed the way you want, it is important to generate your Magento 2 robots.txt file correctly. You don’t have to manually create the file as Magento creates it automatically. You can just add custom instructions to the file and it will be configured accordingly.
If you have any issue in configuring robots.txt file for you shop, then contact our support team for an instant solution.