Robotic - Robots.txt Checker
What is Robotic?
Robotic tests if a web page is protected against downloading by web spiders that respect the robots.txt robots exclusion protocol.
Using Robotic
Select your web spider and then type the address (URL) of the page you wish to test into the form below.
Include the 'https://' or else…
Robotic will determine whether the URL is accessible or protected, according to the server's robots.txt file.