I'm currently working on a gaming site where users will submit virtual players for different games, like Chess, Nash, Backgammon, Go, etc. The idea is that users don't compete themselves, but through their virtual players. There will be leagues, tournaments, and other competition formats.

The question is which would be a good rating system for users in this environment. Take into account that every user may have many different virtual players playing in many different games. As a general guideline I would like to guarantee the following properties:

Users who have a lot of mediocre players should not score higher than users with a few very good players.

A user with a high rating should not be penalized if he adds a new bad player, until he has had enough time to improve his player.

Users who don't play often should not score higher than users who play every day.

Could you elaborate more on differences between a user and a player? Are "virtual players" just an AI playing chess etc. and so You have your army of NPCs playing classic games?
–
Markus von BroadyNov 2 '12 at 22:01

1

There really is no best way to make a scoring system, I would personally probably do something like: base_win_amount + skill_differenace_bonus * (1 - my_skill / opponents_skill). You just have to play with the values a bit, but the idea is that the closer your level is to your opponent the less you will get from the skill bonus, a lower skill level gives you more of the skill bonus, and a higher level causes you to lose points from the base win amount.
–
Benjamin Danger JohnsonNov 2 '12 at 22:18

@MarkusvonBroady It's is exactly like that. You'll have a hoard of NPC players playing in a lot of different games, some playing in a league, others saved for tournaments, perhaps some even playing in teams.
–
Alejandro PiadNov 5 '12 at 16:28

6 Answers
6

Bottom level is where players are rated. Use ELO or a similar system, where a strong player is penalized much more for loosing with a weak opponent, than he could be awarded. Also secure game system for users using many accounts, e.g. by random matchmaking. This will solve your 3rd point:

Users who don't play often should not score higher than users who play every day.

Middle level is rating for every group of players owned by one user and playing one game, e.g. Alejandro's Chess players. You most probably don't want a chess player to play with a Go player, as they use different algorithms, and even the best player of one game will probably lose with one of the worst players in another. To calculate the rating of a group, take an average of ELO of the players in the group, not considering 20% of the worst players in this group. This will ensure the extraordinarily terrible players won't affect average rating. Keep in mind this is yet another reason to make matchmaking random. So this covers two remaining points of your requirements

Users who have a lot of mediocre players should not score higher than users with a few very good players.

A user with a high rating should not be penalized if he adds a new bad player, until he has had enough time to improve his player.

Top level is rating for a user. This is a very subjective part; you could for example:

show best achievement of the user (#23 place in silver league)

show average of scores of all game-groups

show average of scores of all game-groups the user is active in

show the highest of scores of game-groups

calculate the main score in a complex way, e.g. find the highest possible score of a user, being an average of chosen groups plus 100 bonus points for every group taken into an average.

I think this is a quite close to what I want. I accepting this answer, since it covers most of my needs, but many many thanks to @PapaAtHome and other for their help. The only thing I'm still a bit in the dark is about the matchmaking. Is there something complicated or I just arrange matches at random? Thanks a lot.
–
Alejandro PiadNov 6 '12 at 19:24

@AlejandroPiad it may depend on how you create/randomize algorithms for players. Maybe a 1200 ELO player will have absolutely no chance to win with 1600 ELO player - in that case it would be wise not match players with some limited strength difference.
–
Markus von BroadyNov 6 '12 at 21:33

ELO works by matching you with an opponent based on your rating, and then transferring some rating from one player to the other based on the result and the rating difference between the players (i.e. a high rated player beating a low rated player gains very little, but if he lost the game he would lose a lot).

I think that is not going to work well without a separate rating per game for each player. For example if a really good backgammon player decided to learn chess, it would seem strange for their rating to decrease while they are just learning how to play. You'd also need to be able to match them with a chess opponent of the appropriate skill, which a single overall rating can't do effectively.

You could still have an overall rating which is calculated as say the maximum of all other ratings, or maybe the sum of all other ratings if you want to encourage people to play all the games.

I would also strongly consider only allowing one player per user, or alternatively one player per game type. This is because new players need to be awarded an initial ELO rating. If they lose their first game then that rating will drop, and creating a new user and getting a fresh rating would be an easy way to exploit the system.

I'm seriously considering the one-rating-per-game system and then somehow combining these ratings to an overall. Of course in each particular game pairings would be done by the particular rating, and not the overall. Its just like you said, I want to avoid that a good user trying to explore a new game begins loosing a lot of points, because that would discourage people from trying new games. Also I would like to encourage people trying new strategies even on games they already dominate.
–
Alejandro PiadNov 5 '12 at 16:36

It looks like you have three free variables to play with.
The first one is the number of 'mediocre playes versus better players', the second one is 'user rating agains new player success rate' and the third one is 'frequent players versus infrequent players'.

My suggestion: Divide and conquer.
Find an elo-inspred rating for each of the three variables and take a (weighted) average of it. Now you have three problems, but the sum of problems with the three might be smaller than the problems of the whole.

A nice suggestion. Any idea on how to attack any of the three particular problems? Thanks.
–
Alejandro PiadNov 5 '12 at 16:32

@Alejandro Piad: I asume you keep track of what players have done.
–
PapaAtHomeNov 5 '12 at 19:58

1

On Mediocre/better players: The elo system is designed to take care for that. Is some player plays a lot against mediocre players (read: with a substanial lower rating than the player has) he will gain near to nothing. Winning from a weaker player does not gain much for the stronger player. On the other hand, winning from a stronger player gains a lot but up to a maximum level. So, there might be no need for keeping trach of that.
–
PapaAtHomeNov 5 '12 at 20:04

1

On starting players: New players should start with a base rating, say 1000. Over time the elo rating for a new player will middle out to the level that is right for the player. Again, a high elo rated player will not gain a lot from playing a new player but the new player does! So, no need realy to keep track of this as a separate value.
–
PapaAtHomeNov 5 '12 at 20:08

1

Frequent versus infrequent players: Now that's a possible problem area but most likely for the infrequent player only. If he looses some rating than, by the nature of not playing too often, it will be hard to gain back his losses. And the same goes for gaining rating points. You might consider a 'limiting factor' for not playing too often in the sense that an infrequent player cannot gain/loose as much as a frequent player. This way you're also promoting to play frequently, it's a way of gaining rating faster.
–
PapaAtHomeNov 5 '12 at 20:13

I've thought about ranking systems quite a bit and I think it's best to create a new ranking system whenever you have a new project since every time you'll have different parameters and requirements. You could adjust for example parameters of the elo ranking etc. but this will always result in a half-hearted solution. Most game companies create their own rankings & matchmakings for every game.

Of course the already existing rankings are not bad by any means but usually they try to generalize a given scenario which only gives a rough trend. Looking at how other ranking systems work is a good basis for your own ranking but usually creating a ranking system is a complex process which even after implementation requires a lot of tweaking. Thousands or millions of dummy scenarios are required in order to evaluate the quality of a ranking system and often the math behind it is not that simple.

Writing down how the ranking is supposed to work is one of the first steps at creating one, although this might sound simple it is a lot more complex than you might think.

Example:

Users who don't play often should not score higher than users who play every day.

This sounds obvious but on the other hand, how do you quantify often? Is someone who only played one time (scored high points) supposed to have a higher ranking who has mid scores but plays every day? You would have to define a mathematical relation between amount of games/days played with the amount of points. How does this relation look like? Is it supposed to be linear, logarithmic, exponential etc. How will this effect the strategy of the players (will they play less put try to play well in order to get higher points)? Will this behavior have an impact on other aspects of the game (less trained players per user?)?

Every requirement will affect the ranking and thus you have to analyse how they will effect each other (maybe give some higher priorities than others or even ignore certain requirements in some cases). Every ranking system is comparable to a dissertation (depending on how detailed you want to make it)

Ranking

I'm not sure whether I understood your request correctly but to me it sounds like you want to create a ranking for every user based on the success of every player a user has.

My first idea would be to create a ranking for every game (chess, go etc.). Rank every player who plays this game in it and then calculate the overall ranking based on these rankings for every user. Of course this would be very complex since you would have to create a ranking for every game BUT there are lot's of decent rankings for every game. This way the only thing you would have to think about (given you found a ranking for every game) is in which way you want to calculate the overall ranking based on the rankings of every game (still not easy but way easier than creating only one ranking containing all information of every game).

This is what I would recommend to someone who doesn't want to spend too much time with the math behind it, which is still a lot.

Edit:

The advantage of a ranking for every game would be that you could use this ranking for your matchmaking process, but this is another topic.

This is close to what I had imagined, defining (or googling) a ranking system tailored for each game, and then compute the overall ranking. It is this overall ranking that gives me the problems, since I want to avoid people submitting tons of players or playing in all games mediocre just to score higher, but on the other hand how do you compare someone who only plays chess to someone who only plays backgammon? Thanks for your helpful post.
–
Alejandro PiadNov 5 '12 at 16:31

1

Of course you could just sum up all points of every player in order to define your overall ranking but I would define a mapping which benefits higher rankings. Calculate the overall points based on the points/player ratio (how this ratio looks like is up to you, in it's most basic form it would be points divided by all players of a user). Comparing two different games? I wouldn't do it I would give very game/ranking the same priority unless you want to say chess > backgammon. I recommend to study one or two ranking systems (idea + math) this will help you to get a deeper understanding.
–
user22553Nov 5 '12 at 17:12

Before diving into formulae, if you want to create a ranking system, you'll have to come up with a list of specifications and requirements. Decide what the ranking system should accomplish and work from there. Be specific. Requirements that are rock solid can be a great guide, where general guidelines, such as the ones you drew up, may lead to confusion instead.

An example. Alice and Bob both have a few good virtual players, but Alice's are slightly better. Alice then continues to add mediocre players to her set. Eventually, she'll have quite a lot of them. According to rule #1, Alice's score may no longer be higher than Bob's and therefore must go down. Rule #2 forbids this.

Carol joins the party. She has one virtual player and it's terrible, no match for anyone of Bob's players. However, on account of her enthousiastic daily participation and rule #3, she still scores higher than Bob, who only plays once a week.

If you stick to basic limitations instead, even seemingly trivial ones like 'winning increases your score', 'playing often adds to your score'... as long as the set is consistent, it will make the mathematics a lot easier.

Very helpful indeed. I didn't thought of the intrinsic inconsistency of these soft rules I proposed. What I want is nothing extraordinary, just a 'natural' ranking system that pays for better players. I want mostly to encourage diversification (people trying new games without fear to loosing a lot of points), and exploitation (having just one or two very good players per game rather than a lot of mediocre players). I know there is some trade-off between these two ideas, there is where I need your help. Thanks for a very helpful post. +1.
–
Alejandro PiadNov 5 '12 at 16:41

Very helpful. I assume this frequency factor has to do with what I was talking about playing often or not. Any idea on how I can calculate this? Something like your-participations/total-events? +1.
–
Alejandro PiadNov 6 '12 at 19:20

How about keeping a moving average on the last n login dates, take the number of days between the last login dates and the current date and take your moving average over these numbers. If a user is less frequent with logging in this average will go up. You can use some hard limit or a more gradual scale on aplying this average.
–
PapaAtHomeNov 6 '12 at 21:09

One more point on using a frequency factor (or someting alike).
–
PapaAtHomeNov 6 '12 at 21:18

One more point on frequency factors. If you use a system where the the winner goes up just as much as the looser goes down then you have a closed system. With N players, each starting with rating X the total amount of elo points in the system is N * X. Which is a nice feature, ratings will never grow 'over the top'. Using a frequency factor for infrequent players will bleed elo points out of the system! Unless you make some calculations in which two players wil always exchange an equal amount of points, for instance by using a factor of Gain * Winner.FrequencyFactor * Looser.FrequencyFactor.
–
PapaAtHomeNov 6 '12 at 21:25