Internetblog.org.uk Web hosting, Domain names, Dedicated servers
Question: What is a robots.txt file? Answer: The Robot Exclusion Standard allows a website owner to exclude certain parts of a website from being accessed by search engine spiders and other types of automated tools. This is useful if you have something internal or used only for testing that you …