Can't find what you're looking for?
Contact our support team and we'll help you out
What is a robots.txt file?
The robots.txt, also known as the robots exclusion standard or robots exclusion protocol, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or crawled.
What it is used for?