For those that are unfamiliar with Robots.txt, this is a text file that should sit in the root of your websites directory. The Robots.txt (RT) file controls what the search engines can look at and index throughout your website. It’s surprising how often I find clients not even aware that such a file exists. Is the RT file the responsibility of an SEO strategy or something much more?

First off I always recommend that clients have a RT file on their server, even if they want all their pages indexed, it’s just good practice to have this file in place. For me it’s the “I am being good” file, for spiders and search engines, I believe that having such a file in place you get a very miniscule amount of positive value. Based on my experience I have seen websites indexed quicker on a larger number of search engines versus websites that don’t use it.

For larger organisations multi channel marketing is often carried out covering PPC, SEO, Affiliate marketing and Direct Marketing. When such activity is carried out, the tendency for duplicate content and unfriendly landing pages is sometimes an issue. Not to mention when tracking you ideally want your data to be segmented into appropriate channels so mixing a combination can lead to unreliable results.

For example I am a strong believer in PPC landing pages and SEO landing pages being handled differently, essentially you are after the same goal; usually this is involves a conversion of some kind. PPC landing pages work best when kept short, including clear defined actions and some smooth looking graphics. SEO landing pages work best in rankings when there is sufficient content added to this formula. So where a RT file can help is that it can block off an entire directory from being indexed, meaning no content will be duplicated and the results captured will be from two separate marketing channels.

A great resource to learn more about Robots.txt files can be found at robotstxt.org

Some examples on the types of robots.txt files:

Allow everything to be crawled:

User-agent: *

Disallow:

To disallow everything from being indexed:

User-agent: *

Disallow: /

To exclude specific folders from being crawled:

User-agent: *

Disallow: /PPC/

Disallow: /Affiliate/

February 2009 has been an important month for SEO’s around the world as some theories were put mainstream by Aaron Wall over at SEO Book. A recent debate had been brewing concerning brands getting more priority over less established sites in natural search? Well it seems that such an update has happened as this is something that I have been monitoring in the very competitive industry of “travel insurance”.

I work on SEO for a number of clients in the travel insurance industry and I started to notice strange listings appearing in the top 10 results. At first I assumed it was Google testing something out, I researched the competitors to see if they had received a sudden impact of link juice, which they hadn’t. What startled me was that some of these websites had very few links in comparison to the usual “top dogs” of the industry. Added to that the actual content on these pages was not great usually and the listings looked completely out of place an irrelevant.

What I did notice was that the sites were .gov or a brand I recognised through other marketing channels, i.e. TV, direct marketing etc. As it’s probably been a week since this update has occurred (I don’t care what Matt Cutts says, this IS AN UPDATE). The random brand websites making an appearance seems to change daily and only one or two websites actually change on the first page, however pages two and three are moving all over the place.

To give you an example of what I mean about this update, look at the screenshot below, displaying the top 10 UK results for Travel Insurance, I have highlighted the term travel insurance to show which listing doesn’t belong in the results.

Now I’m sure this will calm down a bit over the coming weeks but this makes ranking for the smaller brands and sites that much harder and the bigger boys seem to have it easy – More to follow.

Areas to keep an eye on that I will investigate:

• Brand anchor texts
• Brand mentions
• URL anchor texts
• Domain age
• Brand Universal Search