What is a web crawler?

Home  Dictionary  Web crawler
 

A small program that systematically browses the World Wide Web. Web crawlers are mainly used by search engines to index new websites, of which copies are automatically download and stored in an index; once integrated into such databases websites can be rapidly located the next time a user searches for them, and displayed amongst a list of results.

A web crawler starts by visiting a set of predetermined addresses, analysing its pages, identifying any external links present and adding them to the list of addresses to be visited, thereby perpetuating its work.

Other denominations:
Web crawler

 
Download our trends reports
Consumer Trends Branding Trends See more reports

Other internet
related terms

A

B

C

D

E

F

G

H

I

J

j

K

L

M

N

O

P

Q

R

S

s

S

T

U

V

W

X

Y