Comments on: Drupal SEO – using robots.txt to avoid content duplication

By: Arison

Arison — Mon, 08 Nov 2010 17:05:12 +0000

The same behavior would have been found if those pages had been given the "nofollow" attribute - the googlebot would not have officially visited them, and although the result could appear in the Search Engine Result Page

By: RSS2MYSQL

RSS2MYSQL — Wed, 27 May 2009 00:52:31 +0000

Important to note that unless your content duplication comes off as malicious or scheme-like, there will be no real penalty. The typical penalty for dupe content is to be less favored on the page of question , when compared to the OP(Original Poster). This mean they might get indexed, and you no, or you just ignored.

Google might even look at your overall content relevancy and decide that the content fits better with your own site, and show you favor for the copied content.--- ive heard of this happening, but never experienced it myself.

By: Steven Davies

Steven Davies — Fri, 18 Apr 2008 21:34:53 +0000

If you block *all* of your Drupal paging URLs your search engine traffic might drop.
That is right Drupalzilla.
And if you want to increase it just continuing to use drupal:)

By: Hagrin

Hagrin — Fri, 01 Feb 2008 03:04:10 +0000

Not only can you use robots.txt to prevent content duplication, but you can use your .htaccess file to resolve any canonical domain issues, Google Webmaster Central to pick your preferred domain and the Global Redirect module to get rid of trailing slashes.

Hope that helps!

By: Drupal SEO : A complete beginners step-by-step video tutorial | fiLi's tech

Drupal SEO : A complete beginners step-by-step video tutorial | fiLi's tech — Fri, 21 Dec 2007 11:34:46 +0000

[...] SEO : The top Drupal SEO modules and how to use them“, make sure you’re “Drupal SEO: using robots.txt to avoid content duplication” and add a few more general website SEO [...]

By: Drupalzilla

Drupalzilla — Mon, 20 Aug 2007 18:06:37 +0000

Unless you have custom Views set up, spiders will not be able to find all of your content if you block paginated content.
When you create nodes and promote them to the front page they will paginate like this: http://example.com/node?page=2.  
If you don't promote posts to the front page, you can access them through taxonomy terms, but those are paginated also, for example, http://example/category?page=2.  
So if you block off all paginated pages, search engines will only be able to find the first 10 posts on your home page and the first 10 posts under any taxonomy term.
There are other ways for nodes to be displayed, but they are all paginated with ?page=...
Post #20 in category X will be blocked from search engines unless you have created a specific link to it from another page. I've blocked dynamic pages on a Drupal site as a test (two different sites).
As soon as I ended the experiment, search engine referrals went up significantly. I usually create a custom front page on my Drupal sites, and block /node (which includes all front page pagination). I let crawlers get to the content through taxonomy terms pages because they are listed with other posts on a similar keyword theme.

By: fiLi

fiLi — Mon, 20 Aug 2007 12:57:05 +0000

Drupalzilla - what's in paged URLs that isn't in direct nodes? what kind of unique content do the paginated sections provide?

By: Drupalzilla

Drupalzilla — Mon, 20 Aug 2007 12:49:03 +0000

If you block *all* of your Drupal paging URLs your search engine traffic might drop. Search engine crawlers won't be able to reach all of your content. Better to only selectively block the specific paginated sections that are causing problems...

By: fiLi

fiLi — Mon, 20 Aug 2007 08:00:01 +0000

Dheeraj - robots.txt isn't the only way to prevent from indexing. You can also insert a noindex metatag to the pages you don't wish to index :

meta name="googlebot" content="noindex,nofollow" /

For paging, I believe you can just do "Disallow: */page=* . I suggest you experiment with robots.txt on the Google webmaster tools to see what the right robots.txt is for you to block the pages you don't want to index. Good luck, let me know if that helped.

By: Dheeraj

Dheeraj — Mon, 20 Aug 2007 06:01:11 +0000

How do I stop paging URL from getting indexed? I'm having duplicate content error as i have taxonomy menu for taxonomy navigation. also the content is displayed on the homepage as well as the content link itself. Paging links are like [domain name]/[category name]/[article title]/?page=2 This link is also getting indexed in search engines. Can we write in robot.txt such that it should not crawl the paging links i.e. paging links are not followed. Currently we have patched the pageing to work on POST rather than the GET method.