To prevent undesirable written content within the research indexes, webmasters can instruct spiders to not crawl specified data files or directories throughout the conventional robots.txt file in the root directory in the area. On top of that, a web page might be explicitly excluded from the online search engine's database https://news.rafeeg.ae