In order to avoid undesirable content within the lookup indexes, site owners can instruct spiders never to crawl certain files or directories from the typical robots.txt file in the root directory of your area. Furthermore, a page is often explicitly excluded from a online search engine's database by using a meta tag certain to robots (commonly ).