Tag Archives: search result

Post navigation

When I browsed through marketing brochures of GIS (Geographic Information System) vendors I noticed that the message is quite similar to search analytics. It refers in general to integration of various separate sources into analysis based on geo-visualizations. I have recently seen quite nice and powerful combination of enterprise search and GIS technologies and so I would like to describe it a little bit. Let us start from the basic things.

Search result visualization

It is quite obvious to use a map instead of simple list of results to visualize what was returned for an entered query. This technique is frequently used for plenty of online search applications especially in directory services like yellow pages or real estate web sites. The list of things that are required to do this is pretty short:

– geoloalization of items – it means to assign accurate geo coordinates to location names, addresses, zip codes or whatever expected to be shown in the map; geo localization services are given more less for free by Google or Bing maps.

– backgroud map – this is necessity and also given by Google or Bing; there are also plenty of vendors for more specialized mapping applications

– returned results with geo-coordinates as metadata – to put them in the map

Normally this kind of basic GIS visualisation delivers basic map operations like zooming, panning, different views and additionally some more data like traffic, parks, shops etc. Results are usually pins [Bing] or drops [Google].

Querying / filtering with the map

The step further of integration between search and GIS would be utilizing the map as a tool for definition of search query. One way is to create area of interest that could be drawn in the map as circle, rectangle or polygon. In simple way it could be just the current window view on the map as the area of query. In such an approach full text query is refined to include only results belonging to area defined.

Apart from map all other query refinement tools should be available as well, like date-time sliders or any kind of navigation and fielded queries.

Simple geo-spatial analysis

Sometimes it is important to sort query results by distance from a reference point in order to see all the nearest Chinese restaurant in the neighborhood. I would also categorize as simple geo-spatial analysis grouping of search result into a GIS layers like e.g. density heatmap, hot spots using geographical and other information stored in results metadata etc.

Advanced geo-spatial analysis

More advance query definition and refinement would involve geo-spatial computations. Basing on real needs it could be possible for example to refine search results by an area of sight line from a picked reference point or select filtering areas like those inside specific borders of cities, districts, countries etc.

So the idea is to use relevant output from advanced GIS analysis as an input for query refinement. In this way all the power of GIS can be used to get to the unstructured data through a search process.

What kind of applications do you think could get advantage of search stuffed with really advanced GIS? Looking forward to your comments on this post.

Sometimes we want to update document values in an indexed field more often than other fields. A good solution to this is to use the field type ExternFileField. The ExternalFileField gets values from an external file instead of the index. Such file can easily be changed and update the field after a commit. Hence no documents need to be re-indexed. A field that has ExternalFileField as type is not searchable. The field may currently only be used as a ValueSource in a FunctionQuery.

The external file contains keys and values:

key1=value1
key2=value2

The keys don’t need to be unique.

The name of the external file must be external_<fieldname> or external_<fieldname>.* and must be placed in the index directory.

A new file type of the type ExternalFileField and field must be added to schema.xml.

<fieldType name="file"

keyField="keyField" defVal="1" indexed="false"

stored="false" valType="float" />

<field name="<fieldname>" type="file" />

keyField is the field that contains the keys and <fieldname> contains the values from the external file.

valType defines the value type of the field.

At Findwise we have used this method for a customer where we wanted to show the most visited pages higher up in the search result. These statistics are changing daily for a lot of pages and we don’t want to re-index all these pages every day.

In May I attended An Event Apart in Boston (AEA). AEA is a 2-day (design) conference for people who working with websites and was created by the father of web designJeffrey Zeldman and the CSS guru Eric Meyer. The conference has a broad perspective, dealing with everything from how to write CSS3 and HTML5 to content strategy and graphic design. This post is about an AEA topic brought up by Whitney Hess: Create design principles and use them to establish a philosophy for the user experience.

Hess wants to create universal principals for user experience to communicate a shared understanding amongst team members and customers and to create a basis for an objective evaluation. The principles suggested by Hess are listed below along with examples of how these can relate to search and search user interfaces.

Stay out of people’s way

When you do know what people want stay out of their way

Google knows what to do when people visit their search at Google.com. They get out of the way and make it easy to get things done. The point is not to disturb users with information they do not need, including everything from modal popup windows or to many settings.

Create a hierarchy that matches people’s needs

Give crucial elements the greatest prominence

This means that the most used information should be easy to find and use. A classic example is that on most university webpages – it is almost impossible to find contact details to faculty members or campus address but very easy to find a statement of the school philosophy. But the former is probably what users mostly will try to find.

Limit distractions

This principle means that you should design for consecutive tasks and limit related information to the information you know would help the user with her current task. Don’t include related information in a search user interface just because you can if the information does not add value.

Provide strong information scent

There should be enough information in search results for users to decide if results are relevant. In an e-commerce site this would be the difference between selling and not selling. A search result will not be perceived as cluttered if the correct data is shown.

Provide signposts and cues

Always make it clear how to start a new search, how to apply filters and what kind of actions can be applied to specific search results.

Provide context

Let the user know that there are different kinds of search result. Display thumbnails for pictures and videos or show msn availability in people search.

Use constraints appropriately

Prevent errors before they happen. Query suggestion is a good way as it helps users correct spelling error before they happen. This saves time and frustration for the user.

Make actions reversible

Make it obvious how to removes filters or reset other settings.

Provide feedback

Interaction is a conversation so let the user know when something happens or when the search interface fetches new search results. Never let the user guess what happens.

Make a good first impression

You only have one time to make a first impression. It is therefore important to spend time designing the first impression of any interface. Always aim to make the experience for new users better. This could mean voluntary tutorials or fun and good-looking welcome messages.

So now what?

Are universal principles enough? Probably not. Every project and company is different and need their own principles to identify with. Hess ended her presentation with tips on how to create company principles to complement the universal principles. Maybe there will be future blog posts about creating your own design principles.

Providing spot-on results with good relevance in Enterprise Search solutions is one of the hardest tasks when working with findability. Sure, it is doable to work out a generic model for ranking results based on the organization’s most common requirements on findability in conjunction with available metadata of the information made findable. But is it enough?

The burning question is: How can you ensure that the generic relevance model does not get outdated once the Findability solution has been in use for a month, half a year, a year and the implementation crew is long gone?

Findwise recently released a large Enterprise Findability solution at a customer in the electrical power industry in Sweden. In the project we identified personalized and adaptive relevance as two key requirements for the findability solution to provide real, future-proof value-in-use to a large set of people with fundamentally different roles within the company. This blog post will focus on the latter requirement, adaptiveness: How can we make sure that an Enterprise Findability solution returns search results that become better and better as the solution is used?

Let user behavior improve the behavior of the search tool

The Enterprise Findability solution rolled out at the power company contains two features that, put together, build the foundation of a continuously improving relevance model:

A feature that promotes popular content given a query term – “social relevance”

A feature that continuously changes the relevance model by boosting the relevance of popular documents – “adaptive relevance

Social relevance

Inspired by e-commerce actors on the web, the delivered Enterprise Findability solution uses the logged behavior of its users to promote popular content. When an end-user searches for, e.g. “terawatt hours”, the solution by default offers search results ranked and sorted according to the generic relevance model. This is what any search tool would do. But this solution also uses search logs to promote popular content just as e-commerce sites have been doing for years – “Other people searching for ‘terawatt hours’ viewed ‘Current power production’ (intranet page), ‘Definition of terms in the electrical power industry’ (PDF document)” etc.

By combining the intel of the search logs (where the end-user behavior of an Enterprise Findability solution is constantly collected) and the best bets (editorially provided “sponsored links”) with the regular search result, end-users are presented with a rich set of information answering their original question from different angles. And the best part of it is that the social relevance feature constantly improves as the tool is used. People get better results as time goes by.

Adaptive relevance

In addition to the social relevance feature, the vast amount of real search behavior compiled in the search logs is used for improving the generic relevance model as well. The solution tracks changes in popularity of content and adapts the document-level scores of documents and web pages in the search index accordingly. If a document is accessed often through the search tool, the document will be deemed “more important” and start climbing towards top positions in the search result. And if a previously popular document becomes less popular as time goes by, the document’s impact on the relevance model is decreased. In the end, content that has great importance for a limited amount of time (such as news items and weekly lunch menus) will first peek and then dip in the search index. The search index and the generic relevance model attached to it will stay fresh.

From generic to personalized search experience

This blog post has pinpointed a couple of solutions for a continuously-improving, generic relevance model in an Enterprise Findability solution. Obviously, generic models are generic, i.e. good enough for the many, not perfect for the few. There are great ways to address personalization solving many of the role-based challenges of Enterprise Findability, but let’s leave that to another, future blog post. Stay tuned!

When someone is using the search function on your web site, your web search, it tells you two things. First of all they have a specific need, expressed by their search query. Second, and more importantly he or she wants you to fulfill that need. If users didn’t care where the service was delivered from, they would have gone straight to Google. Hence, the use of your search function signals trust in your capabilities. This means that even if the majority of your website visitors doesn’t use the search function, you know that the ones who do have a commitment to you. Imagine you are working in a store as a clerk; the customer coming up to you and asking you something is probably more interested in doing business with you than the ones just browsing the goods.

This trust however, can easily be turned to frustration and bad will if the web search result is poor and users don’t find what they are looking for. Continuing our analogy with the store, this is much like the experience of looking for a product, wandering around for a few minutes, finally deciding to ask a clerk and getting the answer “If it’s not on the shelf we don’t have it”. I certainly would leave the store and the same applies for a web site. If users fail when browsing and searching, then they will probably leave your site. The consequence is that you might antagonize loyal customers or loose an easy sale. So how do you recognize a bad search function? A good way to start is to look at common search queries and try searching for them yourself. Then start asking a few basic questions such as:

Does the sorting of the search results make sense?

Is it possible to decide which result is interesting based on the information in the result presentation?

Is there any possibility to continue navigating the results if the top hits are not what you are looking for?

Answering these questions yourself will tell you a lot about how your web search is performing. The first step to a good user experience is to know where your challenges are, then you can start making changes to improve the issues you have found in order to make your customers happier. After all, who wants to be the snarky store clerk?

Search patterns are standardized patterns describing search functionality as well as human information seeking behavior. Earlier this year Peter Morville and Jeffery Callender released a book about search patterns. Morville also gave a presentation based on the book at the IA Summit 2010 (slides, mp3), which my colleague Maria and I attended. Among the patterns Peter Morville mentions, my favorite ones are structured and actionable search results.

Structured search results

Let us start with structured search results. You might have seen that for certain queries you submit on Google, you get a richer results presentation than for other results. For example, typing the query ‘weather stockholm’ gives a basic weather forecast for the upcoming four days, directly visible in the results list. Other examples include local movie showtimes and stock information. It is even possible to use google as a calculator or a currency converter by typing in certain kinds of searches. For the curious, here is a list of all google.com search features. Structured results is about offering a more informative presentation of search results than just a title, summary, and possibly some basic metadata. It is also about not presenting all information in the same way, because the information in itself differs. Richer results presentations speeds up the process of finding relevant information since the system has already done some pre-processing for user.

Structured metadata is a prerequisite for structured search results presentation. Web pages and documents normally come with standard metadata such as date and author, but in some cases they will have to be augmented with additional information in order to create a more useful presentation. Presenting results in a custom way requires some extra development effort, especially if the structure is not initially available. However, I believe it creates much value to the user. Also, this need not be done for all types of contents. My advice would be to identify the cases where a more elaborate results presentation would be most usable. Which information is frequently requested by many people and perhaps also difficult to find because it is embedded in pages with lots of text or other contents? Search logs and user feedback in combination with thorough knowledge about the contents provides a key basis for the selection.

Actionable search results

Related to structured results are actionable search results. Entries in the search results list can be more than just displays of information; they can also be means of performing tasks. Common examples found on the web include printing, saving or sharing the search result directly from the results list. Other examples include adding to shopping cart, commenting and rating. Within the enterprise or organization additional relevant actions could perhaps be checking in or out a document, add an event to the personal calendar, starting a chat with a co-worker, and so on. As with structured results, it is about identifying the cases where it would add most value. What are the most common tasks and possibly also what tasks are complicated to perform in the source system? Structured and actionable results share the advantage that users do not have to open the actual results web page or download the document to find or do what they need. Speeding up information seeking and other tasks in this way is not only valuable in web search, it can also be very useful within the enterprise or organization. Search results lists in enterprise search solutions still look quite homogeneous and there are lots of opportunities for improvement.

To conclude, there can and should be more to search results presentation than just a snippet. I believe we will benefit from putting focus on the results presentation, and not only on tools surrounding it (filtering for example). After all, the list of results is where the user’s attention is first drawn. What do you think? How can your organization benefit from working with structured and actionable search results? If you are curious about this approach, we would be happy to help you look into what can be done in your organization.

As noted by Swedish daily paper Metro, Findwise is working with JCDEC, the Joint Concept Development & Experimentation Centre at Swedish Military Headquarters. In Metro’s words the project aims at developing a knowledge management system for the headquarters of tomorrow. The system is expected to be up and running in time for the international military exercise VIKING 11, to be executed in April of 2011.

Good decisions stem from good information; this is true for both military and civilian enterprises. Vast amounts of time and resources are being invested in order to collect information. But to what end? Granted, somewhere among that information there is probably something you will find useful. But large amounts of information quickly become incomprehensible. In order to combat information overload you need a select-and-filter tool such as Search, and that’s where Findwise comes in.

However, for JCDEC it is not enough to simply locate the information they have available. Captain Alexandra Larsson, Concept Development Lead for Knowledge Support, makes this fact very clear. It is just as important to get an idea of what information is not there. In essence, JCDEC is in the process of creating information from information. This is also one of the great differences between the kind of web-based search and retrieval systems we have come to depend on and a state of the art knowledge management system. The latter is not just a retrieval tool; it is an information workbench where the user can select, retrieve, examine and manipulate information.

The key to finding information gaps is to study patterns. For example, consider the trivial problem of birthday distributions. Without any prior knowledge one would probably expect there to be roughly as many births in May as in August or November. This is not always the case. Depending on where you are in the world birth figures may actually be skewed so that one month has significantly more births than other months do. Why does this happen? Being able to pose that exact question may in turn teach us a lot about the mysterious workings of the Stork.

In military intelligence the filling of information gaps may mean the difference between victory and defeat. Why is there an increase in partisan activity in that district? Why were eight weapons silos raided over the course of two days? Why at this moment in time? These questions are expected to lead to insights into the plans and activities of suspects and to notify those in command of looming threats.

Retrieval

The envisioned work-flow for JCDEC information operators is threefold: retrieval, visualization and communication. Each research session will typically be initiated through a keyword search interface, much like you would issue a web query on Google. Just like its online counterpart the system would present the results ordered according to their expected relevancy to the operator’s query. Using facets and query refinement the result set can be narrowed down until the information in front of the operator is expected to contain that which is being sought for.

Captain Alexandra Larsson hints at another strategy for getting to information. Facets are so speedy these days that they can be applied on the full document set without any delays. Clearly, JCDEC is using search technology to provide directory listings much like websites such as the Open Directory Project, although completely dynamic. The option of simply browsing these directories is also available to operators.

Visualization

The next step, visualization, employs an array of tools for visually displaying the results. These include plotting objects on maps and timelines and looking for groupings where objects have a disproportionately dense distribution, so called cluster analysis, among others. This is where clues are uncovered and questions posed: why there, at that time, with those people? In some cases a field investigation is necessary in order to answer these questions. Other times the answers can be deduced from the tools themselves. The tools also allow the operator to formulate new search queries based on the visual information. The operator may choose to limit the scope of the search to one or more of the clusters in the timeline or map, for example.

Communication

If or when the operator finds something interesting this should be recorded. But to JCDEC it is not necessarily the results themselves that are important. The act of getting to the information is valuable in itself. The reason for this is that different operators have different backgrounds and possess different types of information. Where one operator filters or deduces information from a search result in one way, another operator might choose a completely different approach and unveil other clues.

According to Captain Alexandra Larsson it is absolutely necessary that operators share knowledge as well as refinement strategies as part of their work. One of the paradigms that JCDEC is looking to experiment with is social bookmarking along with the ability to search through sets of bookmarked objects. Objects can be both tagged and commented on, useful for conveying meta-information to fellow operators. It is likely that there will be custom-based filters, where an operator can inform the system of types of objects that do and do not interest him or her and have the system automatically filter the result sets based on this information. These filters can of course also be shared with other operators.

An evolving system

The process of retrieval, visualization and communication is only one, albeit the most prominent, feature of the JCDEC knowledge management system. The system itself will be put to use in the spring of 2011 and development will surely continue beyond that point. The ideas and concepts at work today will most likely be refined over time as Captain Alexandra Larsson and JCDEC learn from hands-on experience with working with information. And as evolution progresses I hope to be able to go into more detail on some of the other tidbits.

Analyzing user behaviour is a key ingredient to make a search solutions successful. By using Search Analytics, you gain knowledge of how your users use the search solution and what they expect to find. With this knowledge, simple adjustments such as Key Matches, Synonyms and Query Suggestion can enhance the findability of your search solution.

In addition to this, you can also tune the relevancy by knowing what your users are after. An exciting field in this area is to automate this task, i.e by analyzing what users click on in the search result, the relevancy of the documents it automatically adjusted. Findwise has been looking into this area lately, but there hasn’t been any out-of-the-box functionality for this from any vendor.

Until now.

Two weeks ago Google announced the second major upgrade this year for the Google Search Appliance. Labeled as version 6.2, it brings a lot of new features. The most interesting and innovative one is the Self-Learning Scorer. The self learning scorer analyzes user’s click and behaviour in the search result and use it as input to adjust the relevancy. This means that if a lot of people clicks on the third result, the GSA will boost this document to appear higher up in the result set. So, without you having to do anything, the relevance will increase over time making your Search Solution perform better the more it is used. It’s easy to imagine this will create an upward spiral.

The 6.2 release also delivers improvements regarding security, connectivity, indexing overview and more. To read more about the release, head over to the Google Enterprise Blog.

Today the focus of my sessions have been content indexing and content processing in both SharePoint 2010 and FAST Search for SharePoint. But first I started of with a session covering the new metadata focus in SharePoint 2010.

Metadata is information about the content and is of key importance for making a good search experience. In SharePoint 2010 the focus and this has increased drastically. New features is Term store which a service that can contain taxonomies, folksonomies, social tagging and keywords. Through this feature they make meta data to accessible through out the whole system on all levels of item creation. Having a structured way of working with meta data will drastically increase the quality of the search result.

Now back to the fun technical geek side of these sessions. In SharePoint 2010 Microsoft have introduced a lot of improvements to the indexing side search. First of they have aliened the two versions of search into using the same connectors. Both FAST Search and SharePoint use a common set of connectors and a common way of building new ones. With this you can use systems as BDC to create connectors even from the application SharePoint Designer which is an extremely simple to use application. BDC, which was found in 2007 as well, has though been enveloped with new features like full security support and support for creating connectors in .NET. This making it easy and streamlined to create new connectors for indexing all kinds of systems.

One of the strengths in FAST Search for SharePoint has always been the document processing. This is a feature that SharePoint search is lacking and is probably an important thing why Microsoft bought FAST. In FAST Search for SharePoint they have taken this in to SharePoint to be easily managed and streamlined. Processing as for example entity extraction, lemmatisation and advanced language detection is now done automatically and can be configured through adding for example inclusion/exclusion words for entity extraction straight in UI (can be manged through PowerShell as well).

But what about all those custom pipeline stages that was used to be a large part of en ESP configuration before? This is a function that is not done as before. No python coded pipeline stage can be added however what you now can do is that you add a “extensibility pipeline stage”. This stage can then be configured to call an .NET application with a set of input properties and then a list of returning properties. In this way you can basically do what ever you want with the text content and then do it with the full power of .NET. Some nice side effects of this for us developers is that creating pipeline stages in the past has always been a hussel. Both since it has bin done in python and that testing it had either demanded an hard to setup instance of Eclipse and ESP or to try it out live in ESP. In the new system since it actually are small console applications that is running this can easily be tested stand alone with good debugging through Visual Studio.

Tomorrow is actually the last day of this conference. That they will for me focused on partner events that covers more the sales perspective on all the new things in SharePoint and FAST.

But now its time for some relaxing and then its time for Enterprise Search evening by the pool event.

The people at Findwise are entering vacation mode one after the other. While finishing up my projects before summer vacation I started thinking about what are the important parts of creating a good search experience. So I wanted to give you a few tips before leaving the office for the summer.

Myself and Caroline participated at Business to Buttons in Malmö in June. I met a lot of talented people and had lots of interesting conversations. One of the topics i ended up discussing the most was: Search is just search, right?

A very common opinion amongst designers is that search is just search. You put a search box in the upper right corner and then you’re done. The search engine has thought of everything else, hasn’t it? I found myself arguing about two things that are very close to my heart:

Choosing the righ search platform

Designing a good search experience

Choosing the right platform

There is a difference between search engine platforms. You just don’t go out and by one and think that’s it. “Search is fixed.” It does matter what platform you choose! Depending on your choice you can tune it in different ways to fit your needs. You don’t just install Google or any other platform for that matter, and think your done. If you do, you’re in trouble. As Caroline wrote about in a previous blogpost, most enterprise search projects with problems, have problems that are not related to the platform but to the fact that the organization does not have a strategic way of working with search.

To give you designers and other design interested people a quick start to this subject I recommend listening to a podcast from Adaptive Paths UX week 2007 where Chiara Fox talks about search and interaction design. (You can download the podcast from Itunes store for free.) It will introduce you to some of the basic things to think about when it comes to getting what you want from your search engine.

Designing a good search experience

When designing a good search experience there are lots of things you should think of. But without getting to involved in advanced filters, navigators, query suggestions and other things you first need to fix the basics. Showing relevant information in the search results. One of the most common problems I meet at new customers is search results lists that make it practically impossible for the users to understand what the result is without clicking on it. All search results look the same no matter if they’re documents, web pages, people, applications, or products. The only way for the user to understand what information they can find in the result is by clicking on it. A search application that forces the user to use pogosticking is in no way better than using poor navigation. So first you need to think about what information needs to be displayed about different types of search result. What information is relevant for a document, or for a web page?

To get you started thinking about this I recommend reading the articlefrom UIe about creating good search results. It will introduce you to some of the basics.The article describes web site search. Enterprise search is off course more complex since you have more types of sources but the basic idea is the same: Show the user the information they need.

So that was two recommendations for your reading list this summer (in case there is a rainy day or two).

If you have any question about choosing the right platform or design good search experiences please contact us. More on these topics will also come after the summer.