A robots.txt file restricts access to your site by search engine spiders that crawl the web. The spiders, before they access pages of a site, check to see if a robots.txt file exists that prevents them from accessing certain pages.

You need a robots.txt file only if your site includes content that you don’t want search engines to index. How can you check that the robots.txt file has been correctly set up ?

Fortunately, there is a very useful tool which allows you to do it in an easy way.

Just open http://tool.motoricerca.info/robots-checker.phtml

Enter URL like this – http://www.domain.com/robots.txt and click on “check robots.txt” button.

The contents of robots.txt file will be displayed.