What is robots.txt?

Home  Dictionary  Robots.txt
 

Text file without format that is used to give instructions to Google crawlers or spiders or those of other search engines to inform them of which areas of a website need to be crawled or indexed and which do not.

These files are uploaded to a website to prevent certain parts of that website from appearing in search results, or to avoid the indexing of duplicated content, or to inform Google of its sitemap, including its URL in the file.

 
Download our trends reports
Consumer Trends Branding Trends See more reports

Other internet
related terms

A

B

C

D

E

F

G

H

I

J

j

K

L

M

N

O

P

Q

R

S

s

S

T

U

V

W

X

Y