Post navigation

Over 10,000 attendees from 85 countries, more than 200 sponsors and exhibitors, and over 250 sessions. Besides these impressive numbers, the 2012 SharePoint conference in Las Vegas has also marked the launch of the new version of SharePoint. Findwise was there to learn and is now sharing with you the news about enterprise search in SharePoint 2013.

In the keynote presentation on the first day of the conference, Jared Spataro (Senior Director, SharePoint Product Management at Microsoft) mentions the three big bets made for the SharePoint 2013 product: CLOUD, MOBILE, and SOCIAL. This post tries to provide a brief overview of what these three buzzwords mean for the enterprise search solution in SharePoint 2013. Before reading this, also check out our previous post about search in SharePoint 2013 to get a taste of what’s new in search.

Search in the cloud

While you have probably heard the saying that “the cloud has altered the economics of computing” (Jared Spataro), you might be wondering how to get there. How to go from where you are now to the so-called cloud. The answer for search is that SharePoint 2013 provides a hybrid approach that helps out in this transition. Hybrid search promises to be the bridge between on-premises and the cloud.

The search results from the cloud and those from on-premise can be shown on the same page with the use of the “result blocks”. The result block, new to SharePoint 2013, is a block of results that are individually ranked and are grouped according to a “query rule”. In short, a query rule defines a condition and an action to be fired when the condition is met. With the use of the result blocks, you can display the search results for content coming from the cloud when searching from an on-premises site and the other way around (depending whether you want the search to be one-way or bidirectional), and you can also conditionally enable these result blocks depending on the query (for example, queries matching specific words or regular expressions).

Screenshot from the post Hybrid search of the Microsoft SharePoint Team Blog showing how results from the cloud are integrated in the search results page when the user searches from an on-premises SharePoint 2013 site.

Before making the decision to move to the cloud, it is wise to check the current features availability for both online and on-premise solutions on TechNet.

Mobile devices

With SharePoint 2013, Microsoft has added native mobile apps for Windows, Windows Phone, iPhone, and iPad, and support across different mobile devices (TechNet), which provides access to information and people wherever the users are searching from.

Also important to mention when talking about mobile, is that the improved REST API widens the extensibility options and allows easy development of custom user experiences across different platforms and devices. The search REST API provides access to the keyword query language parameters, and combining this with a bit of JavaScript and HTML allows developers to quickly start building Apps with custom search experiences and making all information available across devices.

Social search

In the same keynote, Jared Spataro said that Microsoft has “integrated social very deeply into the product, creating new experiences that are really designed to help people collaborate more easily and help companies become more agile.” This was also conveyed by the presence of the two founders of the enterprise social network Yammer in the keynote presentation. The new social features integration means that the information about people following content, people following other people, tags, mentions, posts, discussions, are not only searchable but can be used in improving the relevance of the search results and improving the user experience overall. Also, many of the social features are driven by search, such as the recommendations for people or documents to follow.

Whether you are trying to find an answer to a problem to which the solution has already been posted by somebody else, or whether you are trying to find a person with the right expertise through the people search, SharePoint 2013 provides a more robust and richer social search experience than its previous versions. And the possibilities to extend the out-of-the-box capabilities must be very attractive to businesses that are for example looking to combine the social interactivity inside SharePoint with people data stored in other sources (CRM solutions, file shares, time tracking applications, etc).

Stay tuned!

It was indeed an awesome conference, well organized, but most of the times it was hard to decide which presentation to choose from the many good sessions running at the same time. Luckily (or wisely), we had more than one Findwizard on location!

This post is part of our series of reports from the SharePoint 2012 Conference. Keep an eye on the Findability blog for part two of our report from the biggest SharePoint conference of 2012!

Presentation Summary

There is a lot of talk about social, big data, cloud, digital workplace and semantic web. But what about search, is there anything interesting happening within enterprise search and findability? Or is enterprise search dead?

Understanding and utilizing the context of both people and topic (subject) is the future of enterprise search and findability. As we have seen the last few years, the amount of information that is created within organisations and elsewhere is growing exponentially. This makes it harder, day-by-day, to find the information that is relevant at any given moment. By organizing information based on topic, by using text analytics, better metadata, adding user tagging, sentiment analysis etc. it is possible to make findability better. A few examples are mentioned in this blog post series on information flow from 2010. The whole point of findability boils down to improving the information flow and access at any given time. Example on Information Flow from the Intranet of Region Västra Götaland.

In order to make sense of any arbitrary information we as humans usually need the help of someone familiar with the topic to help us makes sense of it and understand it. By both addressing the challenge of finding people with the right knowledge and finding the right information, we can contextually make the information more relevant and easier to find.

For example by doing search analytics and looking at usage patterns in general or by looking at how people with the same usage (search) patterns are going about finding information, we can give better suggestions. Also, recommendations of information produced or liked by people who are like you have a better chance of being relevant to you. By using Social Network Analysis, we should be able to find patterns in what information is in demand and how the informations flows. The analysis can of course also be used to find the supernodes, meaning the people through which information and connections flow. For example, email is a under-utilized source of information flow, knowledge, context and social network analysis.

Last Thursday about 50 of Findwise customers, friends and people from the industry gathered in Stockholm for a Findability day (#findday12). The purpose was simply to share experiences from choosing, implementing and developing search and findability solutions for all types of business and use cases.

Henrik Sunnefeldt, SKF, and Joakim Hallin, SEB, were next on stage and shared their experiences from working with larger search implementations.
Henrik, who is program manager for search at SKF, showed several examples of how search can be applied within an enterprise (intranet, internet, apps, Search-as-a-Service etc) to deliver value to both employees and customers.
As for SEB, Joakim described how SEB has worked actively with search for the past two years. The most popular and successful implementation is a Global People Search. The presentation showed how SEB have changed their way of working; from country specific phone books to a single interface that also contains skills, biographies, tags and more.

During the day we also had the opportunity to listen to three expert presentations about Big data (by Daniel Ling and Magnus Ebbeson), Hydra – a content processing framework – video and presentation (by Joel Westberg) and Better Business, Protection & Revenue (by David Kemp from Autonomy).
As for Big data, there is also a good introduction here on the Findability blog.

Niklas Olsson and Patric Jansson from KTH came on stage at 15:30 and described how they have been running their swift-footed search project during the last year. There are some great learnings from working early with requirements and putting effort into the data quality.

Least, but not last, the day ended with Kristian Norling from Findwise who gave a presentation on the results from the Enterprise Search and Findability Survey. 170 respondents from all over the world filled out the survey during the spring 2012 that showed quite some interesting patterns.
Did you for example know that in many organisations search is owned either by IT (58%) or Communication (29%), that 45% have no specified budget for search and 48% of the participants have less than 1 dedicated person working with search? Furtermore, 44,4% have a search strategy in place or are planning to have one in 2012/13.
The survey results are also discussed in one of the latest UX-postcasts from James Royal-Lawson and Per Axbom.

Thank you to all presenters and participants who contributed to making Findability day 2012 inspiring!

We are currently looking into arranging Findability days in Copenhagen in September, Oslo in October and Stockholm early next spring. If you have ideas (speakers you would like to hear, case studies that you would like insight in etc), please let us know.

Presented by Joel Westberg, Findwise AB
This presentation details the document-processing framework called Hydra that has been developed by Findwise. It is intended as a description of the framework and the problem it aims to solve. We will first discuss the need for scalable document processing, outlining that there is a missing link between the open source chain to bridge the gap between source system and the search engine, then will move on to describe the design goals of Hydra, as well as how it has been implemented to meet those demands on flexibility, robustness and ease of use. This session will end by discussing some of the possibilities that this new pipeline framework can offer, such as freely seamlessly scaling up the solution during peak loads, metadata enrichment as well as proposed integration with Hadoop for Map/Reduce tasks such as page rank calculations.

We now have well over a hundred respondents. The more respondents the better the data will be, so please help spreading the word. We’d love to have several hundred more. The survey will now be open until the end of April.

But most important of all, if you haven’t already, have a cup of coffee and fill in the survey.

A Few Results from the Survey about Enterprise Search

More than 60% say that the amount of searchable content in their organizations today are less or far less than needed. And in three years time 85% say that the amount of searchable content in the organisation will increase och increase significantly.

75% say that it is critical to find the right information to support their organizations business goals and success. But the interesting to note is that over 70% of the respondents say that users don’t know where to find the right information or what to look for – and about 50% of the respondents say that it is not possible to search more than one source of information from a single search query.

In this context it is interesting that the primary goal for using search in organisations (where the answer is imperative or signifact) is to:

Improve re-use of information and/or knowledge) – 59%

Accelerate brokering of people and/or expertise – 55%

Increase collaboration – 60%

Raise awareness of “What We Know” – 57%

and finally to eliminate siloed repositories – 59%

In many organisations search is owned either by IT (60%) or Communication (27%), search has no specified budget (38%) and has less than 1 dedicated person working with search (48%). More than 50% have a search strategy in place or are planning to have one in 2012/13.

These numbers I think are interesting, but definitely need to be segmented and analyzed further. That will of course be done in the report which is due to be ready in June.

The shortest dictionary definition of semantics is: the study of meaning. The more complex explanation of this term would lead to a relationship that maps words, terms and written expressions into common sense and understanding of objects and phenomena in the real world. It is worthy to mention that objects, phenomena and relationships between them are language independent. It means that the same semantic network of concepts can map to multiple languages which is useful in automatic translations or cross-lingual searches.

The approach

In the proposed approach semantics will be modeled as a defined ontology making it possible for the web to “understand” and satisfy the requests and intents of people and machines to use the web content. The ontology is a model that encapsulates knowledge from specific domain and consists of hierarchical structure of classes (taxonomy) that represents concepts of things, phenomena, activities etc. Each concept has a set of attributes that represent the mapping of that particular concept to words and phrases that represents that concepts in written language (as shown at the top of the figure below). Moreover, the proposed ontology model will have horizontal relationships between concepts, e.g. the linguistic relationships (synonymy, homonymy etc.) or domain specific relationships (medicine, law, military, biological, chemical etc.). Such a defined ontology model will be called a Semantic Map and will be used in the proposed search engine. An exemplar part of an enriched ontology of beverages is shown in the figure below. The ontology is enriched, so that the concepts can be easily identified in text using attributes such as the representation of the concept in the written text.

Semantic Map

The Semantic Map is an ontology that is used for bidirectional mapping of textual representation of concepts into a space of their meaning and associations. In this manner, it becomes possible to transform user queries into concepts, ideas and intent that can be matched with indexed set of similar concepts (and their relationships) derived from documents that are returned in a form of result set. Moreover, users will be able to precise and describe their intents using visualized facets of concept taxonomy, concept attributes and horizontal (domain) relationships. The search module will also be able to discover users’ intents based on the history of queries and other relevant factors, e.g. ontological axioms and restrictions. A potentially interesting approach will retrieve additional information regarding the specific user profile from publicly available information available in social portals like Facebook, blog sites etc., as well as in user’s own bookmarks and similar private resources, enabling deeper intent discovery.

Semantic Search Engine

The search engine will be composed of the following components:

Connector – This module will be responsible for acquisition of data from external repositories and pass it to the search engine. The purpose of the connector is also to extract text and relevant metadata from files and external systems and pass it to further processing components.

Parser – This module will be responsible for text processing including activities like: tokenization (breaking text into lexems – words or phrases), lemmatization (normalization of grammar forms), exclusion of stop-words, paragraph and sentence boundary detector. The result of parsing stage is structured text with additional annotations that is passed to semantic Tagger.

Tagger – This module is responsible for adding semantic information for each lexem extracted from the processed text. Technically it refers to addition of identifiers to relevant concepts stored in the Semantic Map for each lexem. Moreover phrases consisting of several words are identified and disambiguation is performed basing on derived contexts. Consider the example illustrated in the figure.

Indexer – This module is responsible for taking all the processed information, transformation and storage into the search index. This module will be enriched with methods of semantic indexing using ontology (semantic map) and language tools.

Search index – The central storage of processed documents (document repository) structured properly to manage full text of the documents, their metadata and all relevant semantic information (document index). The structure is optimized for search performance and accuracy.

Search – This module is responsible for running queries against the search index and retrieval of relevant results. The search algorithms will be enriched to use user intents (complying data privacy) and the prepared Semantic Map to match semantic information stored in the search index.

As we all know the smartphone user base is growing explosively. According to www.statcounter.com, internet access from handheld mobile devices has doubled yearly since 2009 adding up to 8,5 % of all page views globally in January 2012. And mobile users want to be able to do all the same things that they are able to do on their PC. And that includes access to the company’s Enterprise Search solution!

The benefits of the sales force being able to search for vital customer information before a meeting or for field service personnel being able to find documentation quickly are quite obvious. So how can an organization tweak its search solution in order to provide convenient access for the mobile users? And above all, what will it cost?

Well, to answer the last question first: much less than you think. Providing for the mobile user is mainly about creating a new front end/UI. The main bulk of your search solution remains the same; indexing, metadata structure and content publishing, for instance, remain essentially unaffected.

But you do need to provide a quite different UI in order for the user interaction to work smoothly considering the specific characteristics of the mobile client primarily when it comes to screen size/resolution and text input. But the smartphone also has a lot of features that the PC lacks – it is always available and it knows exactly where you are, it always has a camera, microphone, speaker, possibly a magnetometer and accelerometer and of course a touchscreen with motions like pinching and swiping etc. And many of these features can be quite useful as the following examples prove:

Google Mobile App for iPhone: in this app, the iPhone senses when the phone is lifted towards the ear and hence knows when to listen for a search command. Since the phone also knows where the user is, a search for “restaurant” automatically generates hits with restaurants in your vicinity.

Scanning a Barcode or QR-code: scanning a Barcode or QR-code with your phone is another way of entering a search string. An example could be a product in a store where the customer could open a price-search-engine and scan the QR-code of the product and see where the best price is.

As you can see, there are plenty of opportunities for those who want to be creative. But for the most part, the I/O will still be done via the screen. At UX Matters there is a great article by Greg Nudelman describing the considerations when implementing search for mobile clients and suggestions for various design patterns that can be efficient (see http://www.uxmatters.com/mt/archives/2010/04/design-patterns-for-mobile-faceted-search-part-i.php). I have included a brief summary below together with illustrations courtesy of UX Matters. But first, some general considerations for mobile clients:

Use Javascript code to detect what type of device is accessing your search solution and if it is a mobile client you display the mobile interface.

Native App or Mobile Web App: Creating a Mobile Web App is easier and cheaper than creating a native App – for one thing you don’t have to create multiple versions for different OS’s (although you still need to test your solution with different browsers/resolutions). Performance wise there isn’t a big difference between Native Apps and Web Apps and mobile browsers are increasingly gaining access to most of the phones hardware as well.

Authentication: SSO for mobile web applications works the same as for desktop browsers. There are also new solutions currently being launched enabling usage of the company’s existing Active Directory infrastructure. One example is Centrify Directcontrol for Mobile enabling a centralized administration within Active Directory of all device security settings, profiles, certificates and restrictions.

Use HTML5 instead of FLASH: iPhones don’t support FLASH but HTML5 is a very capable alternative

Testing: How the design looks for different resolutions can be tested through various emulators but it is always recommendable to test on a limited set of real smartphones as well.

Access needs to be quick and simple: user interaction is more cumbersome on a phone than on a PC. Normally try to avoid solutions that require more than 3 input actions.

Menu navigation: links on the right side are normally used to drill down in the menu hierarchy and left up/towards the home screen

Gestures: is a very powerful toolbox that can be used in many different ways to create an efficient UI. For example, use “pinch to show more” if you want to expand the summary information of a specific item in the search hit list or “swipe” to expose the metadata (or whatever action you want to assign to that gesture).

Be creative: the mobile client is inherently different from a PC, limited in some ways but more powerful in others. So if you just try to adopt design solutions from the PC and fit them into a mobile UI you are missing out on a lot of powerful design solutions that only make sense on a mobile client and you are definitely not giving the users the best possible search experience. Also, since mobile design is still evolving you don’t need to be limited by conventions and expectations as much as on the PC side – make the most of this freedom to be creative!

W3C mobile: for more information about mobile web development, see http://www.w3.org/Mobile/ which also includes a validating scheme to assess the readiness of content for the mobile web

Mobile faceting can be tricky but by using design patterns like “4 Corners”, “Modal Overlays”, “Watermarks” and “Teaser Design” the UI can become both intuitive and easy to learn as well as providing reasonably powerful functionality. As mentioned, these techniques are summaries from an article written by Greg Nudelman for UX Matters. If you are eager to learn more, feel free to check out Greg’s website and his upcoming workshops focused on mobile design http://www.designcaffeine.com/category/workshops/

4 Corners: instead of stealing scarce real estate by adding faceting options directly on the screen together with the search result, semitransparent buttons are available in each corner enabling the user to bring up a faceting menu by tapping in a corner (see illustration 2).

Modal Overlays: the modal overlay is displayed on top of the original page. The modal overlay works well together with the 4 corners design – tapping a corner opens up the overlay containing faceting functions like filtering and sorting (see illustration 2).

Watermarks: a great technique for guiding users and showing the possibility of using new functions. The watermarks, possibly animated, show a symbol for the available action, for instance arrows indicating that a swiping gesture could be used (see illustration 3).

Persistent Status Bar: always maintain a persistent status bar containing the search string together with applied filters in the search result page. This helps the user maintain orientation. Note that all of the illustrations above have a persistent status bar.

Conclusion

Although Best Practices for mobile UI design are still evolving, plenty of progress has already been made and there are several solutions and design patterns to choose from depending on the specific circumstances at hand. So an implementation project need not be rocket science, as long as you learn the right tricks…

Bringing enterprise information to the field, readily available in a mobile handset or tablet, will mobilize your employees. The UI requires rethinking as we have seen. And security needs to be addressed properly to avoid having sensitive data compromised. But other than that, you are ready to go!

There is a very controversial and highly cited 2006 British Medical Journal (BMJ) article called “Googling for a diagnosis – use of Google as a diagnostic aid: internet based study” which concludes that, for difficult medical diagnostic cases, it is often useful to use Google Search as a tool for finding a diagnosis. Difficult medical cases are often represented by rare diseases, which are diseases with a very low prevalence.

The authors use 26 diagnostic cases published in the New England Journal of Medicine (NEJM) in order to compile a short list of symptoms describing each patient case, and use those keywords as queries for Google. The authors, blinded to the correct disease (a rare diseases in 85% of the cases), select the most ‘prominent’ diagnosis that fits each case. In 58% of the cases they succeed in finding the correct diagnosis.

Several other articles also point to Google as a tool often used by clinicians when searching for medical diagnoses.

But is that so convenient, is that enough, or can this process be easily improved? Indeed, two major advantages for Google are the clinicians’ familiarity with it, and its fresh and extensive index. But how would a vertical search engine with focused and curated content compare to Google when given the task of finding the correct diagnosis for a difficult case?

Well, take an open-source search engine such as Indri, index around 30,000 freely available medical articles describing rare or genetic diseases, use an off-the-shelf retrieval model, and there you have Zebra. In medicine, the term “zebra” is a slang for a surprising diagnosis. In comparison with a search on Google, which often returns results that point to unverified content from blogs or content aggregators, the documents from this vertical search engine are crawled from 10 web resources containing only rare and genetic disease articles, and which are mostly maintained by medical professionals or patient organizations.

Evaluating on a set of 56 queries extracted in a similar manner to the one described above, Zebra easily beats Google. Zebra finds the correct diagnosis in top 20 results in 68% of the cases, while Google succeeds in 32% of them. And this is only the performance of the Zebra with the baseline relevance model — imagine how much more could be done (for example, displaying results as a network of diseases, clustering or even ranking by diseases, or automatic extraction and translation of electronic health record data).

When I browsed through marketing brochures of GIS (Geographic Information System) vendors I noticed that the message is quite similar to search analytics. It refers in general to integration of various separate sources into analysis based on geo-visualizations. I have recently seen quite nice and powerful combination of enterprise search and GIS technologies and so I would like to describe it a little bit. Let us start from the basic things.

Search result visualization

It is quite obvious to use a map instead of simple list of results to visualize what was returned for an entered query. This technique is frequently used for plenty of online search applications especially in directory services like yellow pages or real estate web sites. The list of things that are required to do this is pretty short:

– geoloalization of items – it means to assign accurate geo coordinates to location names, addresses, zip codes or whatever expected to be shown in the map; geo localization services are given more less for free by Google or Bing maps.

– backgroud map – this is necessity and also given by Google or Bing; there are also plenty of vendors for more specialized mapping applications

– returned results with geo-coordinates as metadata – to put them in the map

Normally this kind of basic GIS visualisation delivers basic map operations like zooming, panning, different views and additionally some more data like traffic, parks, shops etc. Results are usually pins [Bing] or drops [Google].

Querying / filtering with the map

The step further of integration between search and GIS would be utilizing the map as a tool for definition of search query. One way is to create area of interest that could be drawn in the map as circle, rectangle or polygon. In simple way it could be just the current window view on the map as the area of query. In such an approach full text query is refined to include only results belonging to area defined.

Apart from map all other query refinement tools should be available as well, like date-time sliders or any kind of navigation and fielded queries.

Simple geo-spatial analysis

Sometimes it is important to sort query results by distance from a reference point in order to see all the nearest Chinese restaurant in the neighborhood. I would also categorize as simple geo-spatial analysis grouping of search result into a GIS layers like e.g. density heatmap, hot spots using geographical and other information stored in results metadata etc.

Advanced geo-spatial analysis

More advance query definition and refinement would involve geo-spatial computations. Basing on real needs it could be possible for example to refine search results by an area of sight line from a picked reference point or select filtering areas like those inside specific borders of cities, districts, countries etc.

So the idea is to use relevant output from advanced GIS analysis as an input for query refinement. In this way all the power of GIS can be used to get to the unstructured data through a search process.

What kind of applications do you think could get advantage of search stuffed with really advanced GIS? Looking forward to your comments on this post.