Chuck Self posted on March 19, 2009 19:24
According to Google, “Duplicate Content” generally refers to substantive blocks of content within or across domains that either completely match other content or are appreciably similar.
If your using DotNetNuke (DNN) there is a very good chance that you have the exact same content that is posted on thousands and thousands of websites.
You might be thinking how?
All you need to do is go look at your Privacy Statement and Terms Of Use pages. These pages will be indexed like any other page on your website and when the search engine compares the text to its database, it will find thousands of pages that are “appreciably similar”.
How can you avoid this duplicate content penalty to your website?
There are a number of ways, but the one I usually opt for is to just use a robots.txt file on your website and block the content from being crawled.
Steps to adding a robots.txt to your DotNetNuke installtion.
1. Create a file using notepad and name it robots.txt
2. Add the following items to the file (assuming FriendlyUrls is on, if not substitute for your own page name).
User-Agent: *
Disallow: /Terms.aspx
Disallow: /Privacy.aspx
Allow: /
3. Save the file and copy it to the root of your DotNetNuke installation folder.
4. Test the file by opening your browser and navigate to www.yourdomain.com/robots.txt.
Please note that this robots.txt file will be applied to all portals on the DNN installation.
One final note, if you use
Google Webmaster Tools (which I highly recommend), you can request to have the
Privacy Statement and
Terms Of Use removed from their index instead of it lingering out until they remove it because it can no longer be crawled.