Mar 12, 2011

Ninjas, data and my start-up

Ted tells me it's not shameless self-promotion if it's also interesting, so here goes nothing. After working on the EQII data set for the past 4 years and generating a lot of research, a few of us on the effort decided to think about commericalizing what we'd learned and thought about along the way. Our effort, Ninja Metrics, debuted at GDC last week and is now working with several companies.

We've developed a series of tools, dashboards and analysis engines to automate the study of large volumes of data. Our coolest tool by far is one that measures social influence and places a dollar value on it. I think it's a big deal. Links, descriptions, gizmos and more below the fold.

Ninja Metrics is the product of my own fevered brain and the expertise of my co-founder, Jaideep Srivastava, a professor at the University of Minnesota. It's also a little bit of a labor of love as long-time TN readers may connect me with a deep love of games, studying players, and playing with data. Together with some of our amazing PhD students, we've invented a tool called the Katana Analytics Engine (Ninjas use katanas. I checked, ok?). What does it do?

The system is designed to work in both subscription and free-to-play environments, and can also be adapted to pure retail or even ad impressions. Really, anywhere where there are social ties and then monetary outcomes. It can predict who will quit, who will spend, who will play more, who’s a gold farmer, and where and how all of these things originated.

The crown jewel of the system, though, is something called the social network influence model. To see it, navigate these tabs on the demo: know your player->churn->network analysis. Zoom in by holding the right mouse button and scrolling up and down.

This dashboard gives the predicted future total value of the entire current network based on deep modeling, i.e. it’s what the whole player base is worth. Then, it attributes a value to every person in the network based on how much they are going to spend plus how much they influence others.

This last part is the real magic: As we know, some people are more influential than others. Knowing who these people are allows a firm to target products or offers to that person so that the influencer will then spread that idea to others. It's like word of mouth, but more powerful and totally known and measurable. It is an entirely different way to think about advertising. We deliver it, and then enable companies to use it to test promotions and interventions, and then see whether they were effective. The system also shows the bottom line increase in sales/network value after the intervention, allowing a marketer to estimate the impact of a campaign for a very large population. Press button here, see increase in value there.

Comments

1.

Trying to montetize 'influence' will lead to treating of the perception of 'influence' as a kind of commodity, and then the 'gaming-of-influence' much like what SEO did to search and how 'comment-spam' became the bastard child of 'blog-comments'. However, with a direct monetary connection this effect innevitability becomes worse. Further, it is likely to innevitably lead to the continued dillution of the real value of genuine social interactions on the web. ~ But hey, it may make a shit-load of money right? ...so that justifies it, right?

A bit cynical, Simon, but logical and worth talking through. Here's why this isn't an issue: the influence and stickiness we model can only come from real human connections, not spoofs of them. For example, a guild leader is influential in a subscription environment. If they stay, others are likely to stay, making them more valuable to the developer than the average player. If they leave, others are likely to leave. Pretty straightforward logic, and I think nothing untoward or unethical.

That guild leader could try to "game" our system (although that seems a bit farfetched to me) by making connections to a bunch of people he/she doesn't really interact with in any meaningful way. However, because those connections are meaningless, they are unlikely to make that person any stickier. So, there would be no increase in influence on retention and the model would recognize it. We don't just measure any links. We measure links that matter over time. Simply increasing the number of them won't lead to any "winning" of a gamed system because we are ultimately detecting real outcomes over time, not fake connections that don't matter.

Dmitri, I understand the proposed logic, however, if you try and make a system that "measures social influence and places a dollar value on it." like many 'semantic web' theories/systems, it would be based on a set of presumptions about patterns of real behviour. However, it is not possible to discriminate between (or model)'benign' influence and (for instance) 'trollesque' behaviour that can stir up enormous traction on the web, but is either less than ethical or (can be) down right evil. If you measure and reward influence by simply tracking 'stickiness' you potentially create incentives (rents) that were not previously present, and may, over time, potentially decrease the real value of what you are apparently trying to measure.

Let me explain how the system works a little more and I think you'll see that the opposite is true. First, we split the player value into two components: first, their own likely future value (say their subscriptions over X months we think they will remain). Second, their influence component, which is them getting others to stay.

That second component works in both directions, positive and negative. So, imagine your troll (I like to call these players asshats, but to each their own). That person enters a social network and is unlikable. Everytime he comes in and shouts "noob" and "fag," players become more likely to leave. Our system sees this and starts attributing their departures to the troll. He is causing a loss by his social influence. So, his influence component is a negative. And if it exceeds his own future value, he gets labeled as a net negative player.

So, if in the unlikely event of a player "gaming" the system, what they would need to do would be to be the kind of person who keeps others around. That is not going to lead to troll-like behavior. Quite the opposite. If anything, the person gaming the system would be acting in a more prosocial way. I'd therefore expect an improvement in behavior overall, not a decline.

Personally, I wouldn't build something that I thought would make the world a worse place. You probably don't know my writings and research, but it's largely focused on community, social capital and improving relationships, and that's reflected in how this algorithm was designed.

I like the concepts behind this, and find myself doing something similar with MMORPG's and how they are critically examined, but agree with Simon. The implications of deliberately quantifying influence and logic have some deep ramifications that basically lead someone to the following question:

Are you the one doing the social engineering, or are your results being socially engineered?

Regardless though I find the potential of the project fascinating -- its the implementation that seems to be the trickiest part.

We'll see if players are actually aware of this or not. It's a little like getting coupons. You know they've figured out something about you, but maybe not why. If they do figure it out, it's figuring out that good behavior leads to better deals. That should mean that the most influential, sticky and helpful people are going to be made happier, and the trolls are disincentived to be there. If that's social engineering, I'm OK with it. But to answer your question it's a little bit of both. We want to work with human nature, and help emphasize the good.

I expect that as companies use these tools there will be experimentation with what works. Some interventions will backfire and some will be very successful. What I will be doing is helping with small tests in part of a network before trying things on a whole one. Happy communities are the most profitable ones, though, so I see this as a win-win.

There have been a few studies that have shown that negative comments and negative behaviour stir much more interest and engagement on the web than positive ones, and I have first hand experience of such phenomena getting ridiculously out of hand. Your principles are fine, but as Edward said, in practice it can prove more tricky.

I think the most vexing problem with effective implementation of something like this in an MMO environment is high levels of communication between players. I'll flesh out the retention example you touched on to elaborate.

Suppose you have a highly influential player whose departure would cause a $100 loss of terminal value. He goes to cancel their account and armed with that $100 NPV a game company can give the player a retention offer to try and keep their business valued at anything less than $100 (though presumably you would take a haircut since attempts to leave the game likely effect long term value of the customer). The player takes the offer and then tells his friends that if you try and quit you will get 20% off subscriptions fees for the next year or whatever the offer he got was. Others players would have different retention segmentations but those that were still eligible for something would likely attempt to cancel in order to see what kind of savings they could get. Full disclosure, I work in the credit card industry, so maybe I'm viewing this through an incorrect lens but perfect information between customers really makes segmentation and differential treatment difficult in retention or loyalty programs.

I do however think that this has incredibly potential to shape the customer experience by targeting influential players for feedback and maybe marketing/acquisition segmentation which would avoid the concerns from other commenter's around player attempts to 'game' influence or NPV metrics to get better treatment. Overall, really cool idea.

Hmmm. It's actually a bit creepy, but I have one thought about this I could add. The players who are being "influential" are acting that way because of other motivations than getting subscription discounts or high-quality buffs. They are doing it because they get off on being influential, per se. Instead of rewarding them in some crass, direct way; reward them by giving them more influence. Maybe with tools they can use in-game to more efficiently manage their contacts and organize them.