What is Robots.txt?
By CleverSage | January 27, 2008
Robots.txt is a text file that tells search engines which sections or pages of your website you don’t want them to visit. It is placed in your root directory and ensures the pages you do not want to be included in search results from appearing. This file is useful not only for keeping your private website information from being seen, but also has benefits in SEO world. Below are a few examples of how to construct a robots.txt file:
This example allows all robots to visit all files because the wildcard “*” specifies all robots:
User-agent: *
Disallow:
This example keeps all robots out:
User-agent: *
Disallow: /
The next is an example that tells all crawlers not to enter into four directories of a website:
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /tmp/
Disallow: /private/
Example that tells a specific crawler not to enter one specific directory:
User-agent: BadBot
Disallow: /private/
Example that tells all crawlers not to enter one specific file:
User-agent: *
Disallow: /directory/file.html
Source: Wikipedia
Implementation
To implement into your site simply open up your website editor or Notepad and place this file in your main directory.
Related Articles & Links
Topics: Accessibility, SEO, Usability, Website Design |