Daily Pakistani Latest Tutorials, TV Serials, Movies, Trailers, Songs and Softwares
How to create a robot file? Why it’s important for your website? How you can create a simple robot text file through which you can tell search engine what they should index. Through robots.txt Website owner gave instruction to Web Robots.
In Robots.txt file we write the code in this way:
The “User-agent: *” means this section will apply to all robots. The “Disallow: /” tells the robots that it should not visit any pages on the site.
You can also prevent directories to index from search engine. Robots will not visit any of your website page from which you will disallow them.
To exclude all robots from the entire server:
To allow all robots complete access:
Or just create an empty robots.txt file, or do not use one at all.
To exclude all robots from part of the server:
To exclude a single robot:
To allow a single robot:
To exclude all files except one:
Instead you can overtly prohibit all disallowed pages:
This was all about robots file. Another way you can tell robots about your sitemap:
This is the way through which you can tell search engines to about your sitemap. In my further advance tutorial about SEO, I will teach you that how you can make your website rank better. You should apply all those techniques which one I am teaching to you. If you have any problem while creating robots.txt file you can tell us. We will help you.