DNN Tips &amp; Tricks #7: DotNetNuke Menu and SEO

Nov18

Written by:Joe Brinkman11/18/2008 12:28 AM

During the OpenForce Conference, I heard from several designers that the DNNMenu had problems with Search Engine Optimization (SEO) because it rendered menu items that used JavaScript for navigation. While I assured them that this was not the case and that the menu rendered a down-level version for search bots, I thought I would perform further testing and document the exact behavior. In the process, I found that while the DNNMenu performed as expected, ASP.Net did not.

To observe the behavior of the menu, I used Firefox 3 with the User Agent Switcher plug-in. This plug-in allows you to simulate any user agent string you wish. For testing purposes you can find a complete list of user agent strings at useragentstring.com.

Below is the DNNMenu as rendered in the standard Firefox browser.

To change user agents, create a set of new user-agent using the Tools>>User Agent Switcher>>Options>>Options screen and add the string for the browser or web crawler you wish to emulate. If you are emulating a web crawler, you should also disable JavaScript. With the GoogleBot user-agent enabled and JavaScript disabled, Firefox will render the menus as seen by Google. You can see this behavior in the picture below.

Opening Firebug, you can easily verify that the menu is rendered as a series of hyperlinks as shown below.

Everything appeared to work as I expected. Unfortunately, when I tested the user-agents for Yahoo and ask.com, I found that the menu did not behave the same as for the Google and MSN/Live bots.

The DNNMenu IsDownLevel property determines whether to render the full JavaScript menu or to render a down-level version that doesn’t require JavaScript. If the “browser” is a web crawler, then the menu should always render as down-level. Looking at the IsDownLevel code below we see that the key is the IsCrawler property which further delegates the crawler check to the ASP.Net framework.

Looking at this code, I see that for some reason ASP.Net is not properly detecting any bots other than GoogleBot and MSN. After doing further research I found that browser detection is controlled by the .Net 2.0 .browser files that are located at %FrameworkDir%\v2.0.50727\CONFIG\Browsers. A quick WinGrep determined that the only browser file that sets the Crawler property is the gateway.browser file. The relevant portion of the file is:

<gatewayid="Crawler"parentID="Default">

<identification>

<userAgentmatch="crawler|Crawler|Googlebot|msnbot"/>

identification>

<capture>

capture>

<capabilities>

<capabilityname="crawler"value="true"/>

capabilities>

gateway>

Clearly, this is only detecting Google and MSN. Further research indicated that this is a known problem with the default ASP.net behavior. ASP.Net 2.0 added the ability to define .browser files in the App_Browsers folder of your web application. By defining custom .browser files we should be able to correctly identify targeted search bots.

As you can see, DotNetNuke has good support for search engines in the default DNNMenu. There are still optimizations you must make if you wish to cater to search engines beyond Google and Windows Live/MSN. One of the great aspects of DotNetNuke is that if you don’t care for the tradeoffs that the DNNMenu makes, then you can choose to use a third-party CSS menu like the HouseMenu or SnapsisCSS NavMenu

All of this works just fine in DotNetNuke 4.4 and beyond, however, in DotNetNuke 5.0 SEO will get a slight boost. The down-level rendering in previous versions of the DNNMenu was not very lightweight nor was did it use semantic markup. The next version of DNNMenu in DotNetNuke 5.0 allows the menu to be rendered as an unordered list which should help search engines to better understand the menu structure. I will have a separate post to discuss this version of the menu.

EDIT: Updated DotNetNuke 4.9.1 and DotNetNuke 5.0.0 to include the Crawler and Webkit browser files from http://owenbrady.net/

Very useful post.
Few questions;
1. Can you put the additional browser files into the %FrameworkDir%\v2.0.50727\CONFIG\Browsers folder so that every app on the server gains the additional detection?
2. You can create a single browser file containing multiple browser definitions can't you?
3. The downlevel URL's generated are they formatted using the configuration of the FriendlyURL provider?
Thanks
antony

By Antony on 11/18/2008 8:53 AM

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

@Will, @Antonio - I'll discuss this with Jon Henning to see if we can come up with a comprehensive file that covers the major bots.
@Antony -
1. I don't know if adding files to that folder will work as those files are compiled and stored in the GAC. You can make changes and compile the new browser files.
2. See above
3. Not sure how the downlevel URLs are generated. I assume that they are normal page links.

Thanks for digging into this Joe.
Why does it have to be so damn complicated though? If we had a menu that emits semantic markup (unordered list), then we would not have to play any kind of user agent tricks at all.

@Tom - I personally agree about the need for a nice simple, CSS based menu. However, as the menu is Jon Henning's baby, he tends to lean more towards providing a richer user experience than what is possible with a pure CSS approach. The good news is that there are already good CSS solutions available, so everyone can choose the menu that they prefer.

Personally, I'm not asking for a pure CSS-driven menu. There is nothing wrong with JavaScript when used intelligently. Telerik's RadMenu is a perfect example ... feature rich AND renders as an unordered list.
Also, you and I and all the geeks above know that the DNN menu follows the provider module and can easily be switched, but that should not be used as an excuse for providing the best possible solutions "out of the DNN box."

Answered my own question re the downlevel URLs. They are generated with the friendly URL provider. Tested with 4.9 with the FriendlyURL's set to 'HumanFriendly' and the downlevel menu displays links like the following:

Joe, thanks for the update, I already looked for ways to get the unordered list working but I did not succeed (it worked in the alpha release way back 2007).
Given the fact that we have default access to jQuery in DNN, we have ample possibilities to skin the menu, including some very nice behaviors. A clean unordered list without additional (menu related) javascript would work fine for me.

@Peter - I tend to agree, but most people using DotNetNuke just want it to work out of the box. I am also sympathetic to Jon's position as he has spent countless hours in development of the existing menu.

I hear you Tom.
I certainly appreciate the time that Jon has put into the existing menu, however I would prefer to see the default be a simple unordered list controlled by CSS / JS if you want. Starting with the son of suckerfish code or equivalent would be a good start.
But, Snapsis is a fabulous menu that I'd recommend to anyone looking for a cleaner, friendly, highly accessible output.

@Joe, don't get me wrong, I truly appreciate the work Jon is doing, without Jon's work, DNN would not be, where it is today. Furthermore, the menu should work out of the box and also the new capabilities for styling are great, there is not much you can't do with nav 2.0. See the great online tutorial at:
http://www.dotnetnuke.com/Community/Blogs/tabid/825/EntryId/1874/New-Video-Available-Demonstrating-New-CSS-Capabilites-in-DNNMenu-v2-0.aspx
I suppose only us "paranoid and UL addicts"-techies (NOM) want to have full control over the menu...
So for those of us who want power, simplicity and easy styling nav 2.0 is clearly the way to go. And for the rest of us, an UnorderListMenuProvider would be a great addition.

Joe, thank you so much for the piece of information. There is a question in my mind after reading your last paragraph. If DNN team switch to unordered list for menu what happens for the menu graphic. As you know, each item would be a simple

tag in which can have only one CSS style.
What happens if there are left, right or top images in the design?
As far as I know the CSS can only apply a background image to the item.
How would they support complex designs?
will the additional contents be injected via CSS styles?
is it an standard way and efficient?
Thanks,
Aboo

By Aboo on 11/26/2008 10:45 AM

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

Aboo,
There are many ways to achieve tabs with rounded corners, etc. with simple UL/LI elements. One popular way is to use the sliding-doors technique by Douglas Bowman. http://www.alistapart.com/articles/slidingdoors/
The technique uses the anchor tag background for the right side, and the surrounding li tag for the left side.

For those interested, I posted on my reasoning for not emitting a UL/LI structure over a year ago here. http://www.dotnetnuke.com/Community/Blogs/tabid/825/EntryId/1511/Next-Version-of-DNNMenu.aspx
It has to do with the amount of markup that gets sent down from the server (raw data will be smaller than data + markup). However, as the blog mentions, it will be possible for the new dnnmenu to emit data + markup by setting the RenderingMode property. Please review that blog entry for more info. Also, to understand the new css capabilities see the entry video mentioned by Peter earlier.

Brendan Kowitz reported a hole in the browser detection in ASP.NET 2.0 back in 2006 and was noted in a recent blog post on Google Webmasters Blog as well. Hard to imagine something so important slippled through the cracks for so long. You can read his original post about it at http://www.kowitz.net/archive/2006/12/11/asp.net-2.0-mozilla-browser-detection-hole.aspx