How to Optimize WordPress Robots.txt Increase SEO

How to Optimize WordPress Robots.txt Increase SEO

 In this article, we will show you how to optimize your WordPress robots.txt for SEO robots.txt file is important for SEO in is help increase search engine ranking.Absence of a robots.txt file will not stop search engines from crawling and indexing any website. However, it is highly recommended that you create one. If you want to submit your site’s XML sitemap to search engines, then this is where search engines will look for your XML sitemap unless you have specified it in Google Webmaster Tools. 

How to Create a Robots.txt file?

Robots.txt file resides in your site’s root folder. You will need to connect to your website using an FTP client/FileZilla, or using cPanel. 

you can open it with a plain text editor like Notepad. All you need to do is create a new text file on your computer and save it as robots.txt. Next, simply upload it to your site’s root folder.

Use Robots.txt file.

robots.txt file is very simple.The first line usually names a user agent. The user agent is actually the name of the search bot you are trying to communicate with. For example, Googlebot or Bingbot. You can use asterisk * to instruct all bots.

The next line follows with Allow or Disallow instructions for search engines, so they know which parts you want them to index, and which ones you don’t want indexed.

here is sample of robots.txt file:

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /readme.html

Adding Your XML Sitemap to Robots.txt File

If you are using Yoast’s WordPress SEO plugin or some other plugin to generate your XML sitemap, then your plugin will automatically add your sitemap related lines into robots.txt file. 

However if it fails, then your plugin will show you the link to your XML Sitemaps which you can add to your robots.txt file manually like this: 
Sitemap: http://www.mysterynav.com/post-sitemap.xml
Sitemap: http://www.mysterynav.com/page-sitemap.xml