WordPress SEO : using robots.txt to avoid content duplication

I tried your suggestion about the robot.txt. I am not sure if it was the reason, but my Google page rank went from a 2 to a 4. The only other reason I can think of is that I moved my blog from the URL http://www.blog.frostfox.com to http://www.frostfox.com/blog.

Frostfox - Yeah, I believe Lonnie is right, but you should know that pagerank only updates once every 3-4 month, so it's not something that you get immediate results on. But redirecting a few reachable urls into one, is very good practice, especially if Google penalized you for duplicate content.

Lonnie - Yeah, there are, and those are all over the SEO blogosphere. You can start off by SEOBook and see his self-report as well as incoming trackbacks. As long as you closely monitor the robots.txt performance in Google webmasters tools, I believe you'll be o'right.

Thanks for the spelling correction :$

Your question about index.php is actually what content duplication is about. Some blogs allow the exact same page to appear through index.php and their main blog path "/" and that's something you want to avoid.

Yeah, I later found your post through various bloggers on the net (JohnTP etc.). That's a good comprehensive post you wrote there...

Glad I could help.

I believe this next plugin will take care of that problem for you (and a few other duplicate content issues) :
Permalink Redirect

Great blog and nice post Fili.. I like how you are keeping it simple, I recently changed my robots.txt from WordPress robots.txt example, to a simpler version, perhaps its time for a followup article???

This robots.txt file looks very simple and thanks for some of the points you made. Now I again have to go through all the site I visited for the robots.txt file and create my own file based on the convincing information provided by them. Hopefully, after experimenting a little I would come to know which is best for a wordpress site. Thank you for the information.

Useful tip and a simple solution. Thanks.

Why does it work for some sites and not others? And how come some blogs get indexed in a day and then are dropped, and others stay in Google indefinitely?…

Some people optimizing their websites/blogs are not aware using of this, it is better that there are always someone who are ready to impart their tips and seo knowledge like you. Thanks for your kindness.

You should check WordPress Robots.txt for Silo SEO as it goes into more detail on removing duplicate content on wordpress using robots.txt.

Ok,.. lets say my blog has been online without a robots file for a couple.. well more than 4 years, traffic comes and goes, but if i change my robots to your recomendations, when google reads it next time, would it remove instantly the duplicate content?

thanks