For those that are unfamiliar with Robots.txt, this is a text file that should sit in the root of your websites directory. The Robots.txt (RT) file controls what the search engines can look at and index throughout your website. It’s surprising how often I find clients not even aware that such a file exists. Is the RT file the responsibility of an SEO strategy or something much more?

First off I always recommend that clients have a RT file on their server, even if they want all their pages indexed, it’s just good practice to have this file in place. For me it’s the “I am being good” file, for spiders and search engines, I believe that having such a file in place you get a very miniscule amount of positive value. Based on my experience I have seen websites indexed quicker on a larger number of search engines versus websites that don’t use it.

For larger organisations multi channel marketing is often carried out covering PPC, SEO, Affiliate marketing and Direct Marketing. When such activity is carried out, the tendency for duplicate content and unfriendly landing pages is sometimes an issue. Not to mention when tracking you ideally want your data to be segmented into appropriate channels so mixing a combination can lead to unreliable results.

For example I am a strong believer in PPC landing pages and SEO landing pages being handled differently, essentially you are after the same goal; usually this is involves a conversion of some kind. PPC landing pages work best when kept short, including clear defined actions and some smooth looking graphics. SEO landing pages work best in rankings when there is sufficient content added to this formula. So where a RT file can help is that it can block off an entire directory from being indexed, meaning no content will be duplicated and the results captured will be from two separate marketing channels.

A great resource to learn more about Robots.txt files can be found at robotstxt.org

Some examples on the types of robots.txt files:

Allow everything to be crawled:

User-agent: *

Disallow:

To disallow everything from being indexed:

User-agent: *

Disallow: /

To exclude specific folders from being crawled:

User-agent: *

Disallow: /PPC/

Disallow: /Affiliate/