It’s that time — the end of another great year. It’s been a phenomenal year for LinkedIn, for my amazing team of data scientists, and for me personally.

The end of year is also an exciting time of transition. I enter 2013 thinking about my daughter starting kindergarten, the house awaiting me after a lifetime of apartment dwelling, and of course all the great things my team is setting out to accomplish.

One of those transitions concerns this blog. About a month ago, I started posting on LinkedIn as an “influencer” — one of a couple of hundred people privileged to use LinkedIn as a native publishing platform. The thousands of people following me there and engaging with my content have convinced me to go all in.

I will keep this site up with the hundreds of posts I’ve published over the past 4+ years, but I’ll put The Noisy Channel on mute for the foreseeable future. So, whether you’re a long-time reader or someone who just got here, I hope you’ll follow me on LinkedIn and read my posts there.

This same week, I’m celebrating my second anniversary at LinkedIn. It’s amazing how quickly two years have gone by, and how the team has grown in numbers and accomplishments. My team post from May already feels so dated! I’ll update it in the next few weeks.

And I just passed 5,000 followers on LinkedIn. I’m gratified that people enjoy my writing, and also thrilled to be playing a part in LinkedIn’s content ecosystem.

I hope all of you are celebrating your own milestones as we approach the holidays and the end of 2012.

I’m curious to hear what you think — both of the content and of the platform. I have no immediate intention to migrate the The Noisy Channel to LinkedIn, but I’m very excited by the initial feedback I’ve gotten for these posts.

I arrived in Maui on Sunday, October 28th, fortunate to miss the “pseunami” warnings prompted by an earthquake off the Canadian coast. And even more fortunate to be thousands of miles away from Hurricane Sandy.

But ironically in this workshop — and throughout the conference — there was more use of Twitter data than of Twitter itself. Most of the tweets using the #cikm2012 hashtag were my own.

DAY 1: USER ENGAGEMENT, EVALUATION BIAS

Tuesday opened with a welcome that included statistics showing how far CIKM has come as a top-tier international conference. There were 1,088 submissions this year! But the highlight of the opening was program co-chair Guy Lebanon demoing software to “improve” paper reviews. It was hilarious, if a bit close to home: the automatically generated reviews looked a lot like those generated by allegedly human reviewers.

We then proceeded to a keynote by Yahoo! Research VP Ricardo Baeza-Yates entitled “User Engagement: The Network Effect Matters!” The title was a bit confusing: it wasn’t about the conventional “network effect“, but rather about user engagement across a network of sites like those owned by Yahoo. He talked about different ways to measure user engagement, and noted that off-site (or, rather, off-network) links ultimately improve users’ downstream engagement. He also observed that style attributes outperform content attributes as predictors of user engagement. Lots of fascinating observations, but I’m curious how well they generalize beyond Yahoo.

I spent the rest of the day making tough choices among the various parallel sessions, starting with the morning session on information retrieval evaluation. Some nuggets from that session: captions and other surface features introduce significant evaluation bias; assessors have poor agreement when evaluating relevance in eDiscovery contexts; and system evaluation improves when it models user differences.

After lunch, I attended a session on web search. Some themes from that session: neighborhood-based methods are effective, whether the neighborhoods are based on document or user similarity; entities and structure are increasingly important for web search. After the coffee break, I went to the social networks session. Topics there included social contagion, online question answering, and social network data anonymization. The talks wrapped up just in time for us to watch the daily cliff diving ceremony before heading to the poster session.

Then back to choosing among parallel sessions. Although I was tempted by the recommender systems session featuring presentations by my LinkedIn colleagues Mitul Tiwari and Bee-Chung Chen, I instead attended the session on ads and products. Two take-aways from that session: ad targeting benefits from explicit identification of user interests; influence maximization can be modeled adversarially as a two-player game.

After the coffee break, I headed to another session on web search — one of my favorite sessions of the conference. There was a talk on query segmentation, a topic responsible for my most popular blog post. Also a great talk on identifying good abandonment, a problem I’ve been interesting ever since hearing about it at SIGIR 2010. Another talk about learning from search logs: generalizing from click entropy to “click pattern entropy” to analyze query ambiguity. And a talk on modeling domain-dependent query reformulation as machine translation using a pseudo-parallel corpus. All in all, a great session packed with practical content.

Then came a purely social evening. The conference reception was a luau, complete with kalua pig, mai tais, hula, and of course poi. Certainly my most memorable conference banquet. I didn’t take pictures, but I recommend Craig Stanfill‘s photos on Flickr.

Thursday began with the last conference keynote: University of Kansas provost Jeffrey Vitter on “Compressed Data Structures with Relevance“. Like the previous keynotes, it was fairly dense, and I suggest you read the papers cited in his abstract if you’re interested in the technical details of how to search for query patterns in massive document collections.

The event started with eBay VP of Research Eric Brill talking about “Having A Great Career in Research”. Unusual in a conference talk, he offered personal and practical advice to students on how to focus their passion and effort towards a happy and successful career. It reminded me of my blog post about dream, fit, and passion, and I hope students took it to heart.

IBM researcher David Carmel gave a talk entitled “Is This Entity Relevant to Your Needs?”. Noting that 71% of web search queries contain named entities (people, places, organizations), he advocated a probabilistic ranking approach to entity-oriented search that ranks retrieved entities according to amount and quality of supporting evidence.

Microsoft Technical Fellow (and former Yahoo! Fellow) Raghu Ramakrishnan talked about “The Future of Information Discovery and Search: Content Optimization, Interactivity, Semantics, and Social Networks”. He packed in a lot of nice material, most of which was from his tenure at Yahoo. He included a nice explanation of explore/exploit, which was also a reminder of how lucky we are at LinkedIn to have hired his former Yahoo colleague Deepak Agarwal.

After lunch, WalmartLabs Chief Scientist AnHai Doan gave a talk entitled “Social Media, Data Integration, and Human Computation”, in which he described constructing a “social genome” by mining social data, connecting it to web data, representing the combined information in a knowledge base. If you’re interested in more details, he’ll be giving an extended version of that talk at LinkedIn on November 29th!

Tencent Research Director Chao Liu talked about “Question Answering through Tencent Open Platform”. Beyond giving a great overview of one of the world’s largest internet platforms, he delivered great self-deprecating lines like “The name is ten cents, and the search engine is soso“.

I spoke next about LinkedIn‘s “Data By The People, For The People“. Given that the talk was right after Halloween and just before the presidential elections, I thought it appropriate to choose a title that would have appealed to one of America’s most distinguished presidents and vampire hunters. If you’re curious to learn more about data science and engineering at LinkedIn (including the publications I cited in my talk), check out http://data.linkedin.com/.

Groupon Director of Research Rajesh Parekh talked about “Leveraging Data to Power Local Commerce”. He focused on a key problem Groupon faces: determining and optimal category mix for each local market. He described how they approach this problem using portfolio theory.

After a coffee break, Adobe Chief Software Architect Tom Malloy talked about “Revolutionizing Digital Marketing with Big Data Analytics”. Apache Pig co-creator — and now Google researcher — Christopher Olston talked about work he did at Yahoo on “Programming and Debugging Large-Scale Data Processing Workflows”. Finally, Microsoft Distinguished Engineer Xuedong (“XD”) Huang gave a talk entitled “From HyperText to HyperTEC”, in which he woke up the audience by having us all participate in the “Bing it On” challenge.

FINAL THOUGHTS

All in all, CIKM 2012 was a great conference in an idyllic setting. Holding the conference in Maui might have been a bit distracting, but the desirability of the location also ensure a high-quality program.

My main complaint is that I don’t like parallel sessions — especially when the topics overlap significantly (e.g., web search sessions competed with those on ranking and recommendations). I’m also not convinced that talks have to be 25 minutes long. Perhaps the conference could more to a format of shorter talks and at least reduce the number of parallel sessions. It would also be great to see more opportunity for interaction — the coffee breaks always felt too short. For more of my thoughts on reforming academic conferences, see my 2009 blog post on the subject.

I also wish more attendees would embrace social media. It’s ironic that researchers who depend so heavily on social media data (especially Twitter) don’t engage in it personally. While I’m honored to have been the conference’s unofficial tweeter (see this visualization of the #cikm2012 tweets), I would have liked to see more attendees engage in a public online conversation. Hopefully others will at least blog about the conference.

But these are quibbles. CIKM continues to be an outstanding conference, and I’m very excited it’s coming to the Bay Area next year. See you at CIKM 2013!

I was fortunate this year not only to be able to attend the 21st ACM International Conference on Information and Knowledge Management (CIKM 2012) in Maui, but also to be invited as part of this year’s industry event, representing LinkedIn.

If you’re attending CIKM, please make sure to say hi to me and Mitul. We’d be delighted to talk to you about the work that we and our colleagues are doing. Our data team also has a site showcasing our team, our publications, and some of the projects we can discuss publicly. And, of course, we’re hiring!

Human-computer information retrieval (HCIR) is the study of information retrieval techniques that integrate human intelligence and algorithmic search to help people explore, understand, and use information. Since 2007, the HCIR Symposium (previously known as the HCIR Workshop) has provided a venue for the theoretical and practical study of HCIR. We even inspired an EuroHCIR workshop across the pond that started in 2011 and is going strong.

Overview

The Sixth Symposium on Human-Computer Interaction and Information Retrieval (HCIR 2012) took place on October 4th and 5th at IBM Research in Cambridge, MA. The 75 attendees represented a cross-section of HCIR research and practice. Over a third of the attendees were from industry — including startups and large technology firms. We had a similar diversity of sponsors, benefiting from the generosity of FXPAL, IBM Research, LinkedIn, Mendeley, Microsoft Research, MIT CSAIL, and Oracle. And we had participants from 6 countries: Canada, Germany, Israel, New Zealand, Switzerland, and the United States.

Keynote

We started the Symposium with a keynote from UC Berkeley professor Marti Hearst, a pioneer in the area of search user interfaces, as well as a prominent researcher of information visualization, natural language processing, and social media analysis. Marti set the tone for the symposium with a visionary keynote that she entitled her “Halloween Cauldron of Ideas for Research”.

She started by talking about the unaddressed seams of sensemaking, reminding us that information seeking is only one part of an overall sensemaking process. She used the challenge of saving and personally organizing search results as an example of a neglected but crucial part of a search interface.

She then challenged us to think about how audio could be used in search interfaces. She cited a study showing that programmers comment their code better when the commenting interface uses speech rather than the keyboard. She then challenged us to consider how auditory notification or feedback could enhance the search experience.

Finally, she presented the idea of “radical collaboration”, offering as an example the use of Mechanical Turk to crowdsource vacation planning. The plans were tested by real tourists, who were delighted with the results.

Marti’s keynote was not only insightful and entertaining (one of her slides featured brain cupcakes!), but notable in how much she engaged all of us in discussion throughout her presentation. This approach was especially appropriate for an HCIR Symposium, given our emphasis on human interaction. For more detail about the keynote, I recommend Gene Golovchinsky’s summary.

Short Paper Presentations

After a coffee break, we had a session devoted to 5 short papers. Each presenter had 10 minutes: 5 minutes to present and 5 minutes for discussion.

We started off with UXLabs director Tony Russell-Rose presenting “Designing for Consumer Search Behaviour“, joint work with University College London researcher Stephann Makri. Tony could not attend in person, so he submitted a video. He presented a framework for describing consumer search behavior along with concrete examples — many of them familiar from the time that Tony and I both worked at Endeca. Most of all, he emphasized the need to close the gap between information science research and industry practice.

Then MIT professor (and Haystack principal investigator) David Karger talked about “Standards Opportunities around Data-Bearing Web Pages“. He argued that there is a small set of standard user interface patterns for authoring structured data: text search, sorting by properties, presenting items in a template, and faceted browsing. He then advocated that these primitives (which have already been implemented in the popular Exhibit framework) be incorporated into a W3C standard so that content authors can use them with the expectation that all modern browsers support them.

Next, Harvard student Elena Agapie presented joint work that she did at FXPAL with Gene Golovchinsky and Pernilla Qvarfordt, entitled “Encouraging Behavior: A Foray into Persuasive Computing“. Information retrieval researchers and practitioners have often argued that longer queries lead to better retrieval performance. But how do we get users to enter longer queries. Elena and colleagues found that the best way was not to explicitly tell them that longer queries are better, but rather to present a halo around the search box that changes color as the query gets longer. A very interesting approach to apply persuasive technology to search!

The short presentation format was extremely effective, encouraging presenters to communicate their ideas efficiently and leaving ample time for discussion.

Posters and Demos

As in previous years, we followed lunch with a vibrant session for posters and demos. Some of the more popular poster themes included question answering, task difficulty, and collaborative information seeking. Here is the full list of poster / demo presentations:

Investigating the Effect of Visualization on User Performance of Information SystemsXiaojun Yuan

Full Paper Presentations

The full paper presentations were split into two sessions, the first held on the 4th and the second held on the 5th. Each presentation slot was 30 minutes. The full papers will be made available soon through the ACM Digital Library.

University of Magdeburg student Marcus Nitsche presented “Knowledge Journey: A Web Search Interface for Young Users”, joint work with Tatiana Gossen and Andreas Nürnberger. The authors performed a study in which they found that children liked having personalized avatars that offer guidance, a wheel-shaped browsing menu, and a coverflow-style results presentation. It will be interesting to see how their study holds up in larger-scale user studies, and whether adults like some of these interface elements too.

University of Waterloo professor Mark Smucker presented joint work with Charlie Clarke on “Modeling User Variance in Time-Biased Gain”. Their simulation-based approach produced distributions of gain that agree with distributions produced by real users. By emphasizing the effect size of differences, their approach could help uncover how much the performance differences among systems matter to real users.

Finally, University of North Carolina at Chapel Hill professor Barbara Wildemuth and University of British Columbia professor Luanne Freund delivered a highly interactive presentation on “Assigning Search Tasks Designed to Elicit Exploratory Search Behaviors”. They performed an extensive survey of information exploration literature to identify concepts that authors have used to characterize exploratory search tasks. They tested examples on the audience to see how well we agreed with their characterization and with one another.

This year, we turned to the problem of people and expertise finding, a topic of obvious personal interest. We are grateful to Mendeley for providing this year’s corpus: a database of over a million researcher profiles with associated metadata including published papers, academic status, disciplines, awards, and more taken from Mendeley’s network of 1.6M+ researchers and 180M+ academic documents.

We asked participants to build systems that could perform three kinds of tasks:

Hiring. Given a job description, produce a set of suitable candidates for the position.

Assembling a Conference Program. Given a conference’s past history, produce a set of suitable candidates for keynotes, program committee members, etc. for the conference.

Finding People to deliver Patent Research or Expert Testimony. Given a patent, produce a set of suitable candidates who could deliver relevant research or expert testimony for use in a trial. These people can be further segmented, e.g., students and other practitioners might be good at the research, while more senior experts might be more credible in high-stakes litigation.

Luanne Freund and Kristof Kessler, both from the University of British Columbia, presented “Exposing and exploring academic expertise with Virtu“, joint work with Michael Huggett and Edie Rasmussen. Virtu takes a task-based approach to expertise, exposing and giving the user control over dimensions of expertise that are more or less desirable depending on the type of expert-finding task. The search interface supports information interaction and exploration through a number of browsing and filtering tools, including facets and sliders. You can try out their application here.

UCLA student Fei Liu presented the “‘iF’ People Search System“, an impressive solo effort. Also unique among the entries, iF is a mobile application, designed for the iPad and supporting swipe and multi-touch gestures. A very slick application, iF offered a novel approach to exploring the corpus of documents and people using the analysis of their reputations and social network relationships.

THE WINNER: Virtu! The competition was fierce, but Virtu stood out for the compelling approach it took to offering users control over the expert-finding process. Congratulations to Luanne, Kristof, and their colleagues for their outstanding work and well-deserved honor.

Reception

After we wrapped up the first day of the Symposium, we walked over to the nearby Technique, a restaurant in the Athenaeum Press building (home to two of Endeca’s offices in our early years) where students of Le Cordon Bleu practice their culinary skills. I’m no master chef, but I certain hope these students earned excellent grades for their performance. We enjoyed a delightful sampling of wines, appetizers, main courses, and desserts.

Conclusion

HCIR has been getting better every year, and this year was no exception. Many attendees in previous years had felt that the one-day format made the event feel rushed, and expanding to a second day took off much of the time pressure. We had ample opportunity for discussion, during the presentations as well as at the coffee breaks and reception. Finally, the Challenge was our best yet, eliciting extraordinary results from the five participating teams.