All You Need To Know About Robots.Txt In SEO

One of the most common yet highly misunderstood terms is dealing with robots.txt in SEO. Moreover, it is a kind of text file that webmaster create to guide web robots (search engine robots) how to crawl web pages on a website.

It is associated with robots exclusion protocol (REP).

REP is a primarily a group of web standards that controls and determines how robots crawl and index web content and present them to the web users.

This protocol also controls web components like meta robots, subdirectory website links like do-follow and no-follow links. Basically, the robots in file indicate whether user agents or web-crawling software can or cannot crawl some parts of the website. Consequently, the crawl instructions can allow or disallow the user agents or customize the permissions.

Here is All You Need to Know About Robots.txt in SEO

Working with it:

The robots file has a basic format as shown:

User-agent: [user-agent name]

Disallow: [URL string not to be crawled]

These two lines count to a complete file. However, a robots file may have multiple lines showing different user agents and instructions viz. allow, disallow, crawl-delay, etc.

In fact, each set of user-agent directives within the robots.txt appear as a discrete set and is separated by a line break.

General robots.txt URL is www.example.com/robots.txt.

Blocking to crawl all the URL on a site using robots.txt.

User-agent: *

Disallow: /

On the other hand, the given code will allow all the web crawlers to access the content on a site.

User-agent: *

Disallow:

Sometimes you need to disallow crawlers from accessing some specific web pages on your site. This can be done as:

User-agent: Googlebot

Disallow: /blogs/blocked-page.html

This will prevent crawling of your web content on the specific URL from Google crawlers.

Where Does Robots.txt Lie on a site?

Web administrator are responsible for managing Robots.txt in SEO.

Whenever a crawler starts crawling your site, they look for the such files. These files lie under www.xyz.com.robots.txt. If they find any instruction about what pages to crawl and what not, they will follow those protocols.

However, if you don’t use a robots.txt file on your site, then web crawlers will crawl all the website content.

When to use Robots.txt file on your Website?

While these are very useful,  it can have a vast impact if you do not use it carefully. Web administrators often end up setting the robots file by disallowing all the web content. This can restrict web crawlers from crawling your entire website.

When used wisely, it can have significant advantages:

• keep your website or specific content private
• avoid duplicate results in SERPs
• Help in specifying the location of the sitemap
• Prevent search engines from indexing files viz. Doc files. Images etc.

Visit : http://www.oodlesmarketing.com/robots-txt-in-seo/