While I’d like to claim credit for the wisdom in this post, alas I cannot. One of Seven Sigma’s consultants (Daniel Wale) worked this one out and I thought that it was blog-worthy. Before I get into the issue and Daniel’s resolution, let me give you a bit of search engine theory 101 with a concept that I find is useful to help understand search optimisation.

Precision vs. recall

Each time a person searches for information, there is an underlying goal or intended outcome. While there has been considerable study of information seeking behaviours in academia and beyond, they boil down to three archetype scenarios.

“I know exactly what I am looking for” – The user has a particular place in mind, either because they visited it in the past or because they assume it exists. This known as known item seeking, but is also referred to as navigationalseeking or refinding.

“I’m not sure what I am looking for but I’ll know it when I find it” – This is known as exploratory seeking and the purpose is to find information assumed to be available. This is characterised by

– Looking for more than one answer

– No expectation of a “right” answer

– Open ended

– Not necessarily knowing much about what is being looking for

– Not being able to articulate what is being looked for

“Gimme gimme gimme!” – A detailed research type search known as exhaustive seeking, leaving no stone unturned in topic exploration. This is characterised by;

– Performing multiple searches

– Expressing what is being looked for in many ways

Now among other things, each of these scenarios would require different search results to meet the information seeking need. For example: If you know what you are looking for, then you would likely prefer a small, highly accurate set of search results that has the desired result at the top of the list. Conversely if you are performing an exploratory or exhaustive search, you would likely prefer a greater number of results since any of them are potentially relevant to you.

In information retrieval, the terms precision and recall are used to measure search efficiency. Google’s Tim Bray put it well when he said “recall measures how well a search system finds what you want and precision measures how well it weeds out what you do not want”. Sometimes recall is just what the doctor ordered, whereas other times, precision is preferred.

The scenario and the issue…

That said, recently, Seven Sigma worked on a knowledgebase project for a large customer contact centre. The vast majority of the users of the system are customer centre operators who deal directly with all customer enquiries and have worked there for a long time. Thus most of the search behaviours are in the known item seeking category as they know the content pretty well – it is just that there is a lot of it. Additionally, picture yourself as one of those operators and then imagine the frustration a failed or time consuming search with an equally frustrated customer on the end of the phone and a growing queue of frustrated callers waiting their turn. In this scenario, search results need to be as precise as possible.

Thus, we invested a lot of time in the search and navigation experience on this project and that investment paid off as the users were very happy with the new system and particularly happy with the search experience. Additionally, we created a mega menu solution to the current navigation that dynamically builds links from knowledgebase article metadata and a managed metadata term set. This was done via the data view web part, XSLT, JavaScript and Marc’s brilliant SPServices. We were very happy with it because there was no server side code at all, yet it was very easy to administer.

So what was the search related issue? In a nutshell, we forgot that the search crawler doesn’t differentiate between your pages content and items in your custom navigation. As a result, we had an issue where searches did not have adequate precision.

To explain the problem, and the resolution, I’ll take a step back and let Daniel continue the story… Take it away Dan…

The knowledgebase that Paul described above contained thousands of articles, and when the search crawler accessed each article page, it also saw the titles of many other articles in the dynamic menu code embedded in the page. As a result, this content also got indexed. When you think about it, the search crawler can’t tell whether content is real content versus when it is a dynamic menu that drops down/slides out when you hover over the menu entry point. The result was that when users searched for any term that appeared in the mega menu, they would get back thousands of results (a match for every page) even when the “actual content” of the page doesn’t contain any references to the searched term.

There is a simple solution however, for controlling what the SharePoint search crawler indexes and what it ignores. SharePoint knows to exclude content that exists inside of <div> HTML tags that have the class noindex added to them. Eg

There is one really important thing to note however. If your <div class=”noindex”> contains a nested <div> tag that doesn’t contain the noindex class, everything inside of this inner <div> tag will be included by the crawler. For example:

In the code above the nested <div> to surround the submenu items does not contain the noindex class. So the text “Article 1.1” and “Article 1.2” will be crawled, while the “Article 1” and “Article 2” text in the parent <div> will still be excluded.

Obviously the example above its greatly simplified and like our solution, your menu is possibly making use of a DataViewWebPart with an XSL transform building it out. It’s inside your XSL where you’ll need to include the <div> with the noindex class because the Web Part will generate its own <div> tags that will encapsulate your menu. (Use the browser Developer Tools and inspect the code that it inserts if you aren’t familiar with the code generated, you’ll find at least one <div> elements that is nested inside any <div class=”noindex”> you put around your web part thinking you were going to stop the custom menu being crawled).

Initially looking around for why our search results were being littered with so many results that seemed irrelevant, I found the way to exclude the custom menu using this method rather easily, I also found a lot of forum posts of people having the same issue but reporting that their use of <div> tags with the noindex class was not working. Some of these posts people had included snippets of their code, each time they had nested <div> tags and were baffled by why their code wasn’t working. I figured most people were having this problem because they simply don’t read the detail in the solutions about the nesting or simply don’t understand that the web part will generate its own HTML into their page and quite likely insert a <div> that surrounds the content they are wanting to hide. As any SharePoint developer quickly finds out a lot of knowledge in SharePoint won’t come from well set out documentation library with lots of code examples that developers get used to with other environments, you need to read blogs (like this one), read forums, talk to colleagues and just build up your own experience until these kinds of gotchas are just known to you. Even the best SharePoint developer can overlook simple things like this and by figuring them out they get that little bit better each time.

Being a SharePoint developer is really about being the master of self-learning, the master of using a search engine to find the knowledge you need and most importantly the master of knowing which information you’re reading is actually going to be helpful and what is going to lead you down the garden path. The MSDN blog post by Mark Arend (http://blogs.msdn.com/b/markarend/archive/2010/06/07/control-search-indexing-crawling-within-a-page-with-noindex.aspx) gives a clear description of the problem and the solution, he also states that it is by design that nested <div> tags are re-evaluated for the noindex class. He also mentions the product team was considering changing this… did this create the confusion for people or was it that they read the first part of the solution and didn’t read the note about nested <div> tags? In any case it’s a vital bit of the solution that it seems a lot of people overlook still.

In case you are wondering, the built in SharePoint navigation menu’s already have the correct <div> tags with the noindex class surrounding them so they aren’t any concern. This problem only exists if you have inserted your own dynamic menu system.

Other Search Provider Considerations

It is more common that you think that some sites do not just use SharePoint Search. The <div class=”noindex”> is a SharePoint specific filter for excluding content within a page, what if you have a Google Search Appliance crawling your site as well? (Yep… we did in this project)

You’re in luck, the Google documents how to exclude content within a page from their search appliance. There are a few different options but the equivalent blanket ignore of the contents between the <div class=”noindex”> tags would be to encapsulate the section between the following two comments

Conclusion

(… and Paul returns to the conversation).

I think Dan has highlighted an easy to overlook implication of custom designing not only navigational content, but really any type of dynamically generated content on a page. While the addition of additional content can make a page itself more intuitive and relevant, consider the implication on the search experience. Since the contextual content will be crawled along with the actual content, sometimes you might end up inadvertently sacrificing precision of search results without realising.

Hope this helps and thanks for reading (and thanks Dan for writing this up)

Note2: Only works with MOSS 2007 sorry as you WSS guys do not have audiences targeting 🙁

This is my small contribution to the SharePoint world. It is a web part that once added to a web part page, allows you to customise the display by adding JavaScript to selectively hide controls on the page . Ever needed to hide a field from display/edit for a certain audience? Well here is a way do it without requiring SharePoint Designer and having to break a page from it’s site definition (unghosting).

Before and after shots below (look ma – no top button!)

To fully understand what is being done here, I suggest you read my series of articles on the use of JavaScript in SharePoint. Part 3 in particular will show you how to safely add this web part to pages with editing disabled (NewForm.aspx, EditForm.aspx and DispForm.aspx)

Kudos to Jeremy Thake for feedback and some code contribution. Despite being seriously metrosexual, he is otherwise otherwise very cool :-P.

Now two important warnings:

Warning 1: This is an alpha quality release and I may never touch it again 🙂 So you very likely *will* break it. If there is enough interest, I am happy to pop it on codeplex

Warning 2: This web part should NOT be considered as a security measure and thus used in any security sensitive scenario (such as an extranet or WCM site). JavaScript by its very nature can be trivially interfered with and thus other methods (server side) should be employed in these scenarios to prevent interference at the browser.

You can download by reading the disclaimer and clicking the button below..

THIS CODE IS PROVIDED UNDER THIS LICENSE ON AN “AS IS” BASIS, WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES THAT THE COVERED CODE IS FREE OF DEFECTS, MERCHANTABLE, FIT FOR A PARTICULAR PURPOSE OR NON-INFRINGING. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE COVERED CODE IS WITH YOU. SHOULD ANY COVERED CODE PROVE DEFECTIVE IN ANY RESPECT, YOU (NOT THE INITIAL DEVELOPER OR ANY OTHER CONTRIBUTOR) ASSUME THE COST OF ANY NECESSARY SERVICING, REPAIR OR CORRECTION. THIS DISCLAIMER OF WARRANTY CONSTITUTES AN ESSENTIAL PART OF THIS LICENSE. NO USE OF ANY COVERED CODE IS AUTHORIZED HEREUNDER EXCEPT UNDER THIS DISCLAIMER

God help me, I’m up to part 6 of series about a technology I dislike and still going. For those of you that have just joined us, then you might want to go back to the very beginning of this series where I used JavaScript to improve the SharePoint user experience. Since then, I’ve been trying to pick a path through the thorny maze of what you could term, ‘sustainable customisation’.

By that, I mean something that hopefully will not cause you grief and heartache the next time a service pack is applied!

So no mood for jokes this time – I want to get this over with so let’s get straight to it and finish this thing!

So where are we at?

Part 1 looked at how we can use JavaScript to deal with the issue of hiding form elements from the user in lists and document libraries.

Part 2 examined some of the issues with the part 1 JavaScript hacks and wrapped it into a web part using the content editor web part.

Part 3 then examined the various issues of adding this new web part to certain SharePoint pages (NewForm.aspx, EditForm.aspx and DispForm.aspx). I also covered using SharePoint Audience targeting to make the hiding/unhiding of form elements personalised to particular groups of users.

Part 4 started to address a couple of remaining usability issues, and introduced ‘proper’ web-part development using Visual Studio and STSDEV. I created a project to perform the same functionality in part 3, but would not requiring the user to have any JavaScript knowledge or experience.

Part 5 then used STSDEV to create a solution package that allowed easy debugging, deployment and updating of the web part developed in part 4.

So what could we possibly have left to cover? Basically this article will revisit the web part code and make some functionality improvements and then I will cover off some remaining quirks/issues that you should be aware of.

Hello and welcome to part 5 of another epic CleverWorkArounds blog post.

If you think I write a lot on my blog, you should see my documentation and training material! I seem to be rare insofar as I actually like to write documentation and can churn out reasonable quality pretty fast. So if you need your scary SharePoint farm/infrastructure audited and fully documented, you know who to call! 🙂

Anyhow, here is the current state of play.

Part 1 of this series looked at how we can use JavaScript to deal with the common request of hiding form elements from the user in lists and document libraries. We looked at a Microsoft documented method, then a better, more flexible method.

Part 2 wrapped this JavaScript code into a web part which has been loaded into the SharePoint web part gallery.

Part 3 then examined the trials and tribulations of getting this new web part added to certain SharePoint pages (NewForm.aspx, EditForm.aspx and DispForm.aspx), and then with a few simple edits, use this web part to hide form fields as desired. Finally, I demonstrated the power of combining this with SharePoint Audiences targeting functionality to make the hiding/unhiding of form elements personalised to particular groups of users.

Part 4 introduced Visual Studio and STSDEV. I created a project to perform the same functionality in part 3, but not requiring any JavaScript knowledge or experience. By the end of part 4 I had a STSDEV project that compiled with no errors.

And now we are onto Part 5 where we turn our attention to the packaging and deployment of our web part. As you are about to see, STSDEV makes this a very quick and painless experience. If you aren’t convinced of the merits of STSDEV and the SharePoint solution framework by the time you finish this article, then I don’t know what will convince you.

Hi there. As I write this post, the media are telling me that the stock market is stuffed, the US economy is going to the dogs and banks are writing down billions from sub-prime excess. I dare not check my online broker, road traffic this morning was abysmal, I was late, brought in the wrong laptop and left an important DVD at home.

Could it get any worse? Who knows, but it sounds like the sort of day to re-visit JavaScript and get frustrated with writing a web part for the first time.

So to recap on our journey thus far..

Part 1 of this series looked at how we can use JavaScript to deal with the common request of hiding form elements from the user in lists and document libraries. It looked at a Microsoft documented method, then a better, more flexible method.

Part 2 wrapped this JavaScript code into a web part which has been loaded into the SharePoint web part gallery.

Part 3 then examined the trials and tribulations of getting this new web part added to certain SharePoint pages (NewForm.aspx, EditForm.aspx and DispForm.aspx), and then with a few simple edits, use this web part to hide form fields as desired. Finally, I demonstrated the power of combining this with SharePoint Audiences targeting functionality to make the hiding/unhiding of form elements personalised to particular groups of users.

All in all a pretty clever workaround so far if I say so myself. 🙂

My original goals for this JavaScript was to find an effective, easily repeatable way to customise SharePoint form pages by hiding fields or form elements when we need to. Specifically:

Allow hidden fields based on identity/audience

Avoid use of SharePoint Designer

Avoid customisations to the form pages that unghosted the pages from the site definition

We achieved these goals in part three, but was I satisfied? No. The quest for more clever workarounds always goes on!

Hey there. Welcome to part 3 of my series on SharePoint customisation using JavaScript and web parts.

So here is the lowdown so far. We are trying to find an effective, repeatable way to easily customise SharePoint form pages, so that we can hide fields or form elements when we need to. The goals were to:

Allow hidden fields based on identity

Avoid use of SharePoint Designer

Avoid customisations to the form pages that unghosted the pages from the site definition

So how have we progressed thus far?.

Part 1 of this series looked at how we can use JavaScript to deal with the common request of hiding form elements from the user in lists and document libraries.

Part 2 wrapped this JavaScript code into a web part which has been loaded into the SharePoint web part gallery.

So let’s knock the rest of this over and pick up right from we left off…

CleverWorkArounds Coffee requirement of this post depends on how much you hate JavaScript.

To quickly recap the first post of this series, we looked at how we can use JavaScript to deal with the common request of hiding form elements from the user in lists and document libraries. The technique demonstrated can be used for columns, buttons and whatever else you want. The method once debugged, is fairly easy to implement with SharePoint designer with and some cut and paste.

But there are several problems with the method that prevent it from getting a better CleverWorkaround rating than “Meh”. They include:

One size fits all, fields are hidden for all visitors irrespective of need.

You need to modify the page in SharePoint designer via cut and paste of JavaScript code

You need to modify auto-generated pages

You need to modify a page from its site definition

Insecure, relying on client side to hide content/controls is not a secure solution

I thought with my last post that involved XSL/XSLT, I’d escape from horrid programming languages and write about more interesting topics but it wasn’t meant to be. This time round I had to delve back into the world of JavaScript – something I swore that I would never do again after a painful encounter back in 2000. (Yep, it’s taken me 8 years to face it again!)

But like everything else with SharePoint, by being a ‘specialist‘, you seem to have to use more technologies and IT disciplines than you would think possible.

As I progressed writing this article, I realised that I was delving back into branding again and toyed with the idea of making this part 8 of the branding series. But the governance topic in part 7 for me rounded off that series of posts nicely, so I will deal with this separately for now and perhaps refresh that series in the future.