Optimize WordPress Robots.txt for SEO

What is robots.txt file?

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It is typically stored in the root directory of your website. 

Recommended robots.txt file for WordPress sites

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/
Sitemap: https://coding17.com/post-sitemap.xml
Sitemap: https://coding17.com/page-sitemap.xml

it is simple to generate robots.txt file.

  1. Go to your website root folder
  2. create robots.txt file
  3. copy above code and paste in the file ( Make sure to change instructions as per your need. )
  4. Change sitemap URL with your generated sitemap URL

Follow this link to know more about robots.txt file.

4 Replies to “Optimize WordPress Robots.txt for SEO”

  1. I do trust all the ideas you have offered on your post.
    They’re really convincing and will certainly work.
    Nonetheless, the posts are very short for novices. May
    just you please prolong them a little from next time?
    Thank you for the post.

  2. This is very interesting, You’re an excessively skilled blogger.
    I have joined your rss feed and sit up for in quest of
    more of your fantastic post. Additionally, I have shared your
    site in my social networks

Comments are closed.