. . . . . . . . .

About Me


"I was thinking about you all day today and what a great person you are."

"I wanted to be #1...After 2 months I reached the top position for my most popular keywords."


Above The Fold!

We Value Privacy

Robots Meta Tags

Each search engine spider wants to index all good content on the web. With that being said, you do not need to tell the search spiders when to revisit or to index or follow. The meta robots tag is standard protocol for robot exclusion. You may have certain pages on your web site that are not fully developed yet, or that you do not want seen for some reason. This is when you would use a robots meta tag.

Meta Robots tag format:

The meta robots tag is the area between the head tags using this context

  • <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> (note: you can use any of the following also. their mission is self explanatory)
  • <META NAME="ROBOTS" CONTENT="INDEX, FOLLOW"> (this tag serves no purpose as it is what the search spiders want to do anyway)

Robots .TXT File

also robots will look for a file called robots.txt in your root directory. if that file exists then they will open it up and look for something along the lines of

this disallows robots to your server this allows complete access
User-agent: *
Disallow: /
User-agent: *

the * indicates all search spiders, and the disallow:/ indicates they should not visit any of your site. rather than using the * you can specify specific robots that you do not want to allow if you only have one that is offending your site. you can also disallow file types or certain directories.

If you have dynamic content that you want a robot to index you should not issue a session ID to the spider.

exclude all robots from parts of site
exclude a single robot allow a single robot disallow specific pages
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /private/
User-agent: BadBot
Disallow: /
User-agent: WebCrawler

User-agent: *
Disallow: /

User-agent: *
Disallow: /~joe/private.html
Disallow: /~joe/foo.html
Disallow: /~joe/bar.html

See Also:

Want Free Keyword Research Software? if yes click here

 Got SEO Questions? Get Answers in Minutes NOT Days
Read the SEO Book blog today for the latest SEO tips.
Search the archives for specific posts
You may also want to ask your questions at the SEO Book community forum.


- Search Engine Marketing Newsletter | Search Engine Marketing Articles | Professional Search Engine Marketing Services | SEO Tips | SEO Tools | PPC Search Engine Tips -
- Search Engines | Web Directories | Search Term Glossary | Site Map | Search Our Directory | Search The Web | Search Engine Marketing Results | Privacy Policy | Link To Us -

- Site FAQ's | Search Marketing Info © 2004 - 2017 | Contact Us -

Filepath: http://www.search-marketing.info /meta-tags/robots-meta.htm
Today is 05/19/24 . This file was last modified on 09/28/17