SEO Terms – Web crawler

Peter Roesler, President - Web Marketing Pros

By Peter Roesler

President, Web Marketing Pros

A Web crawler (or web spider) is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion.  This process is called Web crawling or spidering. Many sites, in particular search engines, use spidering as a means of providing up-to-date data. Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a Web site, such as checking links or validating HTML code.


Share This Article

Top

Contact Us