Custom Robots.txt Generator For Blogger

Custom Robots.txt file generator is a free application to generate SEO
friendly robots.txt file for Google Blogger/Blogspot. 

Example: https://www.compromath.com. Don’t use / at the end of domain name.


How to Add Robots.txt in Blogger

To use this tool, insert your domain name and click on Generate button. You
will see the result. Copy the output code and updated your robots.txt file
from Blogger Dashboard → Settings → Custom robots.txt

To make sure the update, check your website link from browser. 

https://www.yourdomain.com/robots.txt

You should see like this – 


  User-agent: Mediapartners-Google
  Disallow: 
  
  User-agent: *
  Allow: /
  Disallow: /search
  
  Sitemap: https://www.compromath.com/sitemap.xml
  Sitemap: https://www.compromath.com/sitemap-pages.xml

Adjusting robots.txt for Blogger

If you want to allow your blog label or category pages, such as https://www.compromath.com/search/label/Lifestyle, to be indexed by search engines, you need to modify your robots.txt file. Add the following line just after Disallow: /search:

Allow: /search/label/

To allow search query pages, like https://www.compromath.com/search?q=how+to+add+robots+txt, to be indexed by search engines, you should remove the line:

Disallow: /search