Robots txt

Robots txts txt - File which search engine spiders search for to see what the site details are for searching. in a Robots txts text file you can disallow specific or all search engine spiders the ability to index portions or all of your web site. Under your main web site shell you should list the file with the Robots txts txt filename. Robots txts txt and Robots txts meta tags are two ways to control search engine spiders.