What is Robots.txt File

We use robots.txt to give instruction to robots about your site. If you want that some URL of your website not visited by robot then you can use it. Before visiting your Web URL robots checks whether it is allowed to visit or not. If you disallow then robots will not visit any page on the site.


User-agent: *

Disallow: /

Disallow: /~joe/bar.html

Here “User-agent: *” means robots can visit the whole web page.

The “Disallow: /" means robots can’t visit any page on the website

The  “~joe/bar.html: /” means robots can’t visit this page “joe/bar.html”.