Optimize WordPress Robots.txt for SEO

What is robots.txt file?

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It is typically stored in the root directory of your website. 

Recommended robots.txt file for WordPress sites

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/
Sitemap: https://coding17.com/post-sitemap.xml
Sitemap: https://coding17.com/page-sitemap.xml

it is simple to generate robots.txt file.

  1. Go to your website root folder
  2. create robots.txt file
  3. copy above code and paste in the file ( Make sure to change instructions as per your need. )
  4. Change sitemap URL with your generated sitemap URL

Follow this link to know more about robots.txt file.

Leave a Reply

Your email address will not be published. Required fields are marked *