What is robots.txt file?
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It is typically stored in the root directory of your website.
Recommended robots.txt file for WordPress sites
User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Disallow: /readme.html Disallow: /refer/ Sitemap: https://coding17.com/post-sitemap.xml Sitemap: https://coding17.com/page-sitemap.xml
it is simple to generate robots.txt file.
- Go to your website root folder
- create robots.txt file
- copy above code and paste in the file ( Make sure to change instructions as per your need. )
- Change sitemap URL with your generated sitemap URL
Follow this link to know more about robots.txt file.