Search 'spiders' (also called 'robots', 'bots' or 'crawlers') designate the automatic processes that examine a given domain name or URL address to index it or update its listing in a search engine. The spider 'crawls' all linked pages and sometimes any page not specifically exempted from indexing.

To prevent sensitive or outdated directories or pages from being crawled web sites require a file called "robots.txt" to be present at the root of the web site structure. Without it search spiders may index all of your web pages including irrelevant ones or those not intended for public viewing. However because of the way they were conceived "robots.txt" files cannot indicate which files to index, only those NOT to be indexed.

Web site theme
Page titles
Text headings
Repeat words
Keyword as defines in META tag
Description as defines in META tag
Inbound/outbound inks
URL
Media content

How spiders rate the importance of each parameter is unique to each search engine, using their own algorithms and set of rules. If a spider detects discrepancies between those items, such as lack of text appearances of prominent keywords, it may disregard keywords and reference your pages based on repeat words or linked text, significantly affecting your chances of high ranking in search listings. Before submission of a web site we always simulate spider crawling then optimize the pages based on the test results. Finally as spiders leave recognizable traces we can tell you which one came to the site and what pages it crawled.

Until industry-wide standards are set there is no choice but to always be up-to-date with search spider technologies and specifics. ActivRank ensures that your site will be coherent and 100% compliant with all major search spiders.