What does Google Robots.txt do


Robots.txt is basically a method of telling a quest engine "do not come in here, please". When a bot finds a robots.txt file, it can 'browse' it and can duly ignore all the URLs contained inside. Therefore pages among the file do not seem in search results. It isn't a failsafe; robots.txt may be a request for bots to ignore the page, rather than an entire block, however most bots will obey the knowledge found inside the file. When you are readied for the page to be included in a search engine, you just changed your robots.txt file and remove the URL of the designated page.

However, there may be some instances where you have got a web site page you do not wish included in search engine results. For example, you'll be in the process of building a page, and do not wish it listed in search engine results till it is completed. In these instances, you wish to use a file called robots.txt to tell a hunt engine bot to ignore your chosen pages inside your website.

For a look engine to stay their listings up to date, and gift the most correct search engine results, they perform an action referred to as a 'crawl'. This is essentially sending a 'bot' (generally referred to as a 'spider') out to crawl the internet. The bot can then find new pages, updated pages or pages it failed to previously understand to exist. The end results of the crawl is that the search engine results page is updated, and every one of the pages found on the last crawl are now included. It's merely a method of finding sites on the internet.
If you have enjoyed this entry. Please feel free to bookmark it using your favorite social bookmarking site

Post a Comment