One of the most common yet highly misunderstood terms is dealing with robots.txt in SEO. Moreover, it is a kind of text file that webmaster create to guide web robots (search engine robots) how to crawl web pages on a website.
Here is All You Need to Know About Robots.txt in SEO
Working with it:
Where Does Robots.txt Lie on a site?
When to use Robots.txt file on your Website?
- keep your website or specific content private
- avoid duplicate results in SERPs
- Help in specifying the location of the sitemap
- Prevent search engines from indexing files viz. Doc files. Images etc.