Robots.txt file is necessary for the visibility of your blog/website on various search engines and one of the most important aspect of best SEO (Search engine optimization) practices. Robots.txt file works to guide Search engine robots or Crawlers how to crawl the website, what to crawl from website? Because when crawlers or robots enters your website, it first reads Robots.txt file, and work according to the instruction (restrictions and permissions). Even you can restrict the robots to crawl particular file, webpage or forler via this text file. Further explaining how to write robots.txt file for most renowned search engines like Google, Bing, Yahoo.
Now the question is How Robots.txt file varies according to different search engines?
Robots.txt file is a standard which is followed by all the crawlers of various search engines. But for better result in different search engines you need to make slightly little changes in the standard robots.txt file, The standard robots.txt file is shown below
The above shown format is the example of standard robots.txt file for Google search engine. But if you want to make it global for Yahoo and Bing search engine as well Site map path declaration is must in Robots.txt file. otherwise Bing and Yahoo will not index your site pages. Finally you have to upload this Robots.txt file in the root folder of the web directory and submit the URL of this file to the different search engine.
No comments:
Post a Comment