Thursday, 24 December 2015

What is Robots.txt File


We use robots.txt to give instruction to robots about your site. If you want that some URL of your website not visited by robot then you can use it. Before visiting your Web URL robots checks whether it is allowed to visit or not. If you disallow then robots will not visit any page on the site.


User-agent: *


Disallow: /


Disallow: /~joe/bar.html


Here “User-agent: *” means robots can visit the whole web page.


The “Disallow: /" means robots can’t visit any page on the website


The  “~joe/bar.html: /” means robots can’t visit this page “joe/bar.html”.



Related Posts:

  • Types of SEO   In the age of Internet Marketing let us see how many types of SEOs are used. There are three different types of SEO: White Hat SEO Black… Read More
  • Meta Tags in SEO Meta Tags A Meta Tag can be described as snippets that describe the overall page content. It doesn’t appear on the web page, but only in … Read More
  • Classified Site List Classified Site List These days everybody wants to promote their website and publishing your website on classifieds sites is one of the best option… Read More
  • Video Promotion Strategy I am going to tell you some strategy to promote your video so that most of the people can view your video. Social Media Profiles- You can cre… Read More
  • SEO Techniques There are two types of SEO techniques. On Page Optimization Off page optimization On Page Optimization In on-Page optimization Technique… Read More

0 comments:

Post a Comment