Robots.txt is often a file that exists on the basis directory of each Web-site and can be used to instruct search engines like yahoo on which directories/information of the website they are able to crawl and include within their index. Each and every time you use a online search engine, https://news.rafeeg.ae