One of the most head-scratching tech headlines of April 2011 was the news that Kosmix, a Mountain View, CA-based startup best known for building a Twitter filtering tool called TweetBeat, had been acquired by Walmart. Yes, that Walmart—the one with 9,000 big-box stores spread across the American heartland.

For one thing, Walmart already has a large technology presence right here in the Bay Area: you can see the big “Walmart.com” sign on the e-commerce division’s building from Highway 101 in Brisbane. So it wasn’t clear why the company needed a second Silicon Valley redoubt. Even more puzzling, Kosmix’s so-called “social genome” platform, which the company had been applying in areas like news aggregation and categorization, didn’t seem to have much to do with Walmart’s business problems—such as narrowing the gap with e-commerce market leader Amazon, for example.

There was speculation that Walmart’s real interest was in Kosmix’s founders, Venky Harinarayan and Anand Rajaraman, who have unbeatable pedigrees in the world of e-commerce technology. The pioneering comparison shopping site they co-founded in 1996, Junglee, was acquired by Amazon in 1998 for $250 million; inside Amazon, the pair helped to create the e-retailer’s huge marketplace of third-party retailers and came up with the technology behind Amazon Mechanical Turk. Perhaps Walmart—which paid $300 million for Kosmix, according to AllThingsD’s Kara Swisher—wanted Harinarayan and Rajaraman to work similar miracles for Walmart.com?

Those were the questions on my mind when I drove down to the former Kosmix headquarters, now WalmartLabs, in Mountain View a couple of weeks ago. I talked for about hour with Rajaraman, who now shares the title of senior vice president of Walmart Global eCommerce with Harinarayan; he’s also an active Silicon Valley investor and writes about his big technology passion, data mining, at a blog called Datawocky. It turned out to be the most extensive interview that either Kosmix founder has given since the acquisition, and I learned a lot about why Walmart thought Kosmix was interesting, and what kinds of capabilities Rajaraman thinks his 70-person team can bring to their new employer.

A lot of it has to do with unsurprising things like improving the product recommendations that Internet users get when they go to Walmart.com, and tapping shoppers’ smartphones as a marketing channel. But Rajaraman also pointed to some more interesting applications for Kosmix’s social genome technology—like monitoring social media conversations in the vicinity of a physical Walmart store for signals about what goods that store should stock.

But we’ll have to wait a bit longer to see what concrete products, features, or campaigns emerge from the Kosmix acquisition. Rajaraman said his team is hard at work on some features that will likely make their debut before the 2011 holidays. He dropped heavy hints that smartphone apps and an enhanced presence for Walmart on Facebook will figure in the changes somehow, but stayed largely mum about the specifics. “In six to eight months the impact is going to be visible, for sure,” he said.

Anand Rajaraman: Walmart is the biggest retailer in the world, but they are not the number-one player in e-commerce—Amazon is. About a year ago, Walmart decided that e-commerce is a strategic priority. It’s not like they had not been investing in e-commerce, but they said, ‘It’s time to go to the next level.’

When you do that, what’s important is to look at how the world has changed. Are there some assumptions that can be challenged, or some trends that can be used, to leapfrog the 800-pound gorilla in e-commerce?

If you think about the way the world has changed in the last two years, there are two big, disruptive changes that have happened, and one of them is social networking. People are spending more time on Twitter and Facebook and the like. And the other is smartphones. For the first time this year, more smartphones were sold in the US than feature phones.

If you put these two things together, they will be as disruptive to retailing as the advent of e-commerce was 15 years ago. The biggest disruptive change in the last century was the development of the highway system, which led to big-box retailing. Then came the invention of the Web. And the third disruption is social and mobile. In each case, the way people shop was changed. The goal of Walmart Labs is to make sure that Walmart is at the forefront of “e-commerce 2.0,” so that we help define it rather than playing catch-up.

X: Why do you think Walmart was attracted to acquiring Kosmix, specifically, as the nucleus for WalmartLabs?

AR: It’s a combination of things. The first is the platform we are building. The fundamental technology we were building at Kosmix is called semantic analysis. We understand the meaning of things. If somebody tweeted “I enjoyed Salt,” we would know that it was a movie with Angelina Jolie and not a food. We are applying semantic analysis to social media and trying to understand the connections between people, topics, places, and products.

We map that space, and we call it the “social genome.” We were using it to operate the Tweetbeat site, where you could find out the pulse of what was going on in social media. But if you look at the founders and management team of Kosmix, we have significant e-commerce experience, and it was pretty obvious to us that the social genome we were building had serious applications to e-commerce.

If you think about the evolution of e-commerce, Amazon did a lot of things right, but the key was using the data they gathered about customers to improve the customer experience. Telling you “People who bought this product also bought these other products”—things like that. Still, there are two significant limitations. One is that Amazon learns about users only by what they do on-site. The products I purchase are a very small window into me, and sometimes a misleading window. Whereas social media gives a much broader window. If you can, with the user’s permission, understand more about what people are passionate about, you can market to them much more accurately.

The second insight is that we can do this anytime if we put an app on their smartphone. When they walk into a Walmart store, we could tell them, ‘Hey, here is a product that we think you will be interested in.” It’s the combination of social and mobile with the Kosmix semantic analysis technology that was the attractive thing for Walmart.

X: Okay, now let me turn the question around. Why would Kosmix want to be part of Walmart? Why would a relatively small, nimble team of Silicon Valley innovators want to work for one of the largest companies in the world?

AR: What really motivates any technologist is the opportunity to build products that are used by hundreds of millions of people and make a big impact. The thing about Walmart is that we get that opportunity. We have this really big canvas to paint on. Any product we build will instantaneously be used by tens of millions of people.

X: But in a way, it still surprises me that a bunch of startup guys like yourselves would want to be part of Walmart, which, just by virtue of its size, has got to be a pretty bureaucratic place.

AR: You’d be surprised. Walmart has been one of the most innovative companies—they practically invented big box retailing, after all. They’ve made huge innovations around the supply chain and merchandising. I teach a class on data mining at Stanford, and interestingly, one of the examples we talk about is from Walmart, which was a pioneer in that space. Perhaps the one place where they didn’t innovate as fast as other companies was e-commerce where they clearly were not the leaders. But it would be wrong to say they do not innovate.