How to create the Robots.txt file for SEO?

robots.txt

What is a robots.txt file?

Robots Exclusion Protocol is also known as robots.txt Protocol.
If you don’t want to index any content or site like “Admin Panel” or “About us” page then you need a robots.txt file.
You can restrict the search engine crawler to indexing the page which you don’t want to crawl on search engine.

What is crawler?

A crawler is a search engine program which visits the websites and read all pages, contents, images and videos automatically to indexing all website.

How to use it?

  • Suppose you don’t want to index the Admin Panel on your website.
    You can write:

User-agent: *

Disallow: /Admin Panel/

  • If you don’t want to index “Contact Us” page then you can put the URL link of Contact Us page and write according to the order:

User-agent: *

Disallow: /Admin Panel/

Disallow: /www.domain-name.com/contact-us/

sitemap: http://www.domain-name.com/sitemap.xml

  • If you specifically don’t want to crawl on “Google search” engine then you can write as follows:

User-agent: Googlebot

For “Bing” search engine:

User-agent: Bingbot

  • After creating the robots.txt file you can place it on the root directory of your website or hosting account.  Then go to “File Manager” and then go to Public_html file and add upload the robot.txt filerobots.txt

 

  • Or you can add the Meta tag to your header file of the webpage which you don’t want to crawl.

<meta name=”robots” content=”noindex, nofollow”>

How to create Robots.txt in WordPress?

You can use the above steps for robots.txt in WordPress also. You can add the meta tag to your header file or you can follow these steps,

User-agent: *

Disallow: /wp-admin/

Disallow: wp-*

sitemap: http://www.domain-name.com/sitemap.xml

  • Login to your WordPress.

robots.txt

  • Click on SEO.

robots.txt

  • Click on File Editor.
  • If your File editor tool looking disable then you have to enable the File editor.

robots.txt

  • Make changes your robots.txt file or you can add.

robots.txt image

Why do you need robots.txt file?

  • Prevent duplicate content.
  • Keeping your Admin Panel safe and secured.
  • Preventing search engine from indexing certain files on your website.
  • Specify the location of the sitemap.
  • Stop crawler to index any specific webpage.

Checking if you have robots.txt file?

Not sure your website has the robots.txt file or not? Simply type your domain name/robots.txt. 

Example: www.bloggingglobe.com/robots.txt or you can search on robot checker.

3 Comments

  1. My husband and i got quite more than happy that Albert could do his preliminary research out of the ideas he acquired from your very own web site. It is now and again perplexing to simply choose to be offering helpful tips which many people might have been selling. And we also do know we have got the website owner to be grateful to for this. Those illustrations you’ve made, the simple blog menu, the friendships you can make it easier to foster – it’s many fabulous, and it’s really leading our son in addition to the family do think the matter is enjoyable, which is really essential. Many thanks for all the pieces!

    1. Thank you for your spending time on our blog. We write all the blogs in simple and helpful for all. We are giving our 100% effort to solve the problem, Please read our blog and bookmarking our blog.

  2. Thank you for every other informative site.The place else could I get that type of information written in such a perfect means?I’ve a undertaking that I am simply now running on, and I’ve been at the glance out for suchinfo.

Leave a Reply

Your email address will not be published. Required fields are marked *