. . . . . . . . .

About Me


"I was thinking about you all day today and what a great person you are."

"I wanted to be #1...After 2 months I reached the top position for my most popular keywords."


Above The Fold!

We Value Privacy

Search Engine Spiders

What are Spiders?

Spiders can be thought of as minion or informants. Web spiders are released onto a few popular sites and from there they follow links absorbing a copy of content in their path.

Spidering Schedule

The internet is starting to get so large that it is hard to spider the entire net more than about once or twice a month. For that reason, spiders are used to regularly update the sites and pages that are viewed as more important while the rest of the web is updated every month or two. Pages which are known to change frequently (such as blogs, forums, and news sites) are spidered frequently, sometimes multiple times a day. The spiders insert this data into a catalog for further processing.

Spider Roadblocks: Dynamic Programming & Java

The spiders are smart and dumb. They are very good at following text links and any old regular hyper link. These same spiders struggle to list complex dynamic pages (good at following simple dynamic pages) and struggle following java script links (Google has been testing a spider that spiders JavaScript).

Spider Precautions With Dynamic Content

If spiders tried to spider a site that issued session ID's or cookies the site can appear much larger than it is to the spider, the spiders can get stuck in an endless loop, or the spiders can run through dynamic content so fast that they crash the server. For that reason spiders are instructed not to follow deep dynamic links without taking pauses between page requests. You still can list dynamic pages easily though.

While search engines now list dynamic content they view everything the record as HTML and it is easy to think of a search engine spider as a HTML photographer.

Want Free Keyword Research Software? if yes click here

 Got SEO Questions? Get Answers in Minutes NOT Days
Read the SEO Book blog today for the latest SEO tips.
Search the archives for specific posts
You may also want to ask your questions at the SEO Book community forum.


- Search Engine Marketing Newsletter | Search Engine Marketing Articles | Professional Search Engine Marketing Services | SEO Tips | SEO Tools | PPC Search Engine Tips -
- Search Engines | Web Directories | Search Term Glossary | Site Map | Search Our Directory | Search The Web | Search Engine Marketing Results | Privacy Policy | Link To Us -

- Site FAQ's | Search Marketing Info © 2004 - 2017 | Contact Us -

Filepath: http://www.search-marketing.info /search-engines-work/spiders.htm
Today is 11/29/23 . This file was last modified on 05/11/13