Sunday, January 6, 2008

The new scoring has been up for a week now, long enough to take a look and see how it's doing.

In general, I like it. I think it's an improvement. First of all, it fixes the "can't play newbies" problem that dogged the previous scoring method. That really hurt me, because it discouraged teaching the game to new players. As always, my goal is to make the game accessible to new players and encourage its spread across Second Life.

Watching the list fluctuate, I think it's pretty close to a good "mixing" level. The names and ranks change, so it's fluid, but it's not chaotic. There's a balance to be had there. I don't want the high scores to be so chaotic that they completely change with every game. But at the same time, I don't want them to be static, with the same names always in the same places -- that would discourage new players. What I'm seeing now is, I think, pretty good.

But, of course, it's not perfect. Here are some questions that have been asked me online and in the comments:

It's too complicated! I don't understand how my score works!Yes, I know. This was the biggest concern I had in implementing this system. I've tried to explain it as best I can in the blog posts below. I hope that, for most players, you don't really have to understand it; you just have to know that if you play well and beat a lot of people, you'll get a high score.

Why can't I see my ranking after I play a game? If I'm not on the board, I can't tell what my score is.I'm sorry, I had to take that out temporarily. The reason is, it takes a long time to compute the new scores, and if the game waited around to give you that new number it could hang up my whole server. The next release of En Garde will tell you your score/rank at the start of the game (when you step on the piste) so you'll see it there, and you'll know the rank of whoever you're playing against.

Why does it only track your scores for two weeks?There's two reasons. First, it adds fluidity to the board. I wouldn't want someone to get a super-high score, and then leave SL, and have their name stand there forever. The board could completely freeze up if that happened enough. So I put in a 2-week time limit. I figure, if you haven't played at all in the last two weeks, then you don't deserve to be on the list.

Secondly, the calculation is expensive (see above), and it gets more expensive the more games it takes into account. Right now there are about 500 games of En Garde played every week. So, after every game, it has to scan through 1000 games and adjust everyone's score.

Have I mentioned that the Procyon Games server runs on a rather low-end system? Well, the hardware is fast, but to save money I have to share it with many other people who have also purchased space on the server, so every CPU cycle counts. I've optimized and re-optimized the scoring system, trying to squeeze every last ounce of processing power I can, but I'm at the limit. Trying to calculate scores over a larger window would make the whole system grind to a halt.

My score can go up with no effort on my part!Yes. If you beat someone, and then that person beats a lot of other people, your score will go up.

Here is how I think about it. How many points should you get for beating someone? Under the old system, the number of points you got was based on their ranking at that time. If they were ranked high, you got a lot of points. It's simple, but it only looked backward.

The new system says: when you beat someone, you get points depending on how good a player they are. And that definition of "good" can change over the 2-week window. If you beat a new player, the system doesn't know if he's good or not. So you don't get many points. Under the old system, it stops there. But under the new system, if that player turns out to be some sort of prodigy, then their rank will improve -- and you will be awarded more points for having beaten them.

PageRank can be gamed. You can't rely on it.Any kind of scoring system can be gamed. My old scoring systems were gamed. Google wages a constant war against Search Engine Optimization companies. My move to this scoring is in part a response to gaming of the previous score systems. At some point, hopefully, we'll reach an equilibrium. Google has an army of software engineers on their side. I've just got me.

Logarithmic scoring?Yes, PageRank is logarithmic. And so is En Garde scoring. This is a technical detail; it happens behind the scenes. The log scale tend to flatten out the numbers. If I didn't have a log scale, then the top players would have scores in the thousands, while people just below them would have scores less than ten. I've tried to adjust the scales so everything "looks right".

Dangling links?In the PageRank algorithm, pages that have no incoming links are a problem, because every page is fed the ranking scores from all the pages that link to it. In EnGardeRank, every player feeds ranking scores to all the other players that have winning records against him. Google adjusts PageRank with a "damping" factor, which represents the random chance a user will move to another page without clicking a link (via bookmark, for example). EnGardeRank is adjusted by your overall score. In the two-week window, it adds your score in every game, and the scores of all your opponents, and that ratio is used to feed every player some small ranking. This represents the random chance you might beat someone that you've never met. So, even a player who hasn't defeated anyone will have a non-zero ranking score.

7
comments:

I appreciate the explanation. But there should still be a way to factor in history so a player doesn't lose everything they've done every two weeks. The suggestion that one does not "deserve" a place in the rankings because they haven't played in so short an interval as two weeks is silly. (Perhaps a divisor that automatically scales down a rating after 14 days inactivity, based on elapsed time since last match, would be possible. I don't know the intricacies of your formula.)

I'm glad to hear about the new version announcing pre-match ratings. It's not terribly important whether folks learn how they're doing pre-match or post-match, but it is vital to the growth of the system that everyone be able to find out how they are doing somehow.

I share your view on the two-week rolling cut-off. What is this ranking - really? It's a snapshot of the best players, currently. Sure, players may slip off, due to inactivity, but just as easy regain their claim for fame.

And, more important in my opinion, it doesn't make the climb up there even more steep for newcomers!

Hello Rifkin - here comes another feature request... although I can offer to code it for you (database and php) if you think it is worthwhile.

Two things really and they depend on accessing your database.

1. A list of players belonging to a particular team - I am considering setting up something that would collect the information but it would be so much better if it could simply draw directly on your data.

2. A list where you can see the complete scoring for individual players - even if just a periodic historical snapshot.

Item one could even be included into #2.

Would you be game to add and host that if I provided php scripts that queries your database?

Rifkin, PLEASE set something up that will allow team owners to see a list of their members. I have people who tell me they've joined the team, but I had no idea. And I've invited people to join who tell me they already have (embarrassing to not know). I hate someone asking me why they haven't been invited to play in any of our get-togethers when I didn't even know they had signed up for my team! It is becoming impossible for those of us, who are setting up matches, to know who to invite. If we are going to foster team play and promote the game, we need to access to this simple bit of information as soon as possible. PLEASE! ~ Thanks, Clair

Well, three weeks into the new scoring and I like what I'm seeing on the World Champion board. (Except that I had to take down all my boards temporarily because they were not properly returning scores. Sigh.) There seems to be both motion amongst players and a fluctuating range of scores.

I will be a happy camper when your announced version that gives pre-match rankings to both players becomes reality. I do miss hearing how I'm doing. :)

As an organiser I second the calls for two features: 1) A way to find out who is on each team, or at least our own team. 2) A way to get a full ranking list of all players once in a while, either as a periodic dump at stated intervals or by request just before a tournament for seeding.

Frankly, I would pay for the full ranking list feature, and I think there are other organisers that might do so.

Keep up the good work--and I'm still looking forward to buying some post-holiday Frootcake. ;)