A Report System Experiment #1 (May 13th - May 27th)

We felt this did not need to be posted earlier because of the report system shaking some of its' problems loose, but it seems that the system really is as broken as we thought. So here it goes!

A small group of friends of mine have done an experiment on the wonderful report system and have found valve to be extremely misleading when it comes to "facts" about this new system.

What prompted this experiment was that the report system was infuriating and causing my group of friends to lose all interest in Dota 2 due to the fear of being muted for illegitimate reasons, no communication ban appeals, extensive passive aggressive trolling by giving power to the masses, & overall lack of communication borne from the broken system that is killing public ques.

People are now AFRAID to talk to each other because of the rampant misuse of the report system, and it makes any que other than one with 3+ friends a huge gamble on if you will be communication banned or not, typically for the smallest of reasons.

Now, let us talk numbers... Really think about how much 1% of the current player base is.

1% of 3,400,000 is 34,000. By valve's estimates 34,000 people are currently under communication ban.

How many truly "unique" players are there in 3,400,000? To give benefit of the doubt, I am going to assume only a couple hundred thousand are mule accounts, remakes/fresh starts, secondaries due to mutes, etc.

Now how many of those unique players play more than 1-2 matches a week? Maybe a million and a half?

If 34,000 is not a lot of people (even out of 1.5-2 million) then my sense of volume is incredibly skewed. Think about how many of those reports are for legitimate purposes, from what I have seen in game maybe 20% of them are LEGITIMATE reports. By that estimate it is 27,200 people that are under communication ban for the wrong reasons.

If these communication bans were seen as a infectious disease or illness, it would be considered an epidemic.

34,000 people infected in less than a week. Where is the cure? Where is the appeals forum or system?

The total amount of reports has dropped by 30%, each player only has 2 reports per week now. Compared to the 7 per week that you had prior to the change of the system.

That means MORE people are being reprimanded now than ever before, but there are less reports because they only allow us to make 2? It doesn't make much sense, especially after looking at the forums.

The article feels like a slap to the face to anyone that feels like digging in to their numbers.

THE EXPERIMENT:
May 13th - May 27th

We have taken numbers of muted or players refusing to speak in our games from a period of 2 weeks. Every single one of us was using an account clean of communication bans or reports, All Pick and Single Draft modes, both Regular and Solo matchmaking on US West or East, varied our play times, and most importantly ONLY USED THE CHAT WHEEL FOR COMMUNICATION.

The 34 of us averaged together 13 games per week per player (889 games total, potentially 8001 different players encountered, we used 7500 for calculations).

The "skill level" of the players differentiate and many are typically in different matchmaking tiers than others along with being able to vary in the middle of the sampling, at the start of the experiment 3 players were consistently Very High Tier, 12 were High Tier, 18 were Normal, and 1 was Low-Priority.

What we found with our small sampling was very strange and seemed to tell us a different story than the article that was posted on the Dota 2 site on May 28th, here is some of what we found:

2377 Players were either Muted or Refused to communicate other than use of chat wheel (31.69% of all players encountered)

223 Threats of reports made against Experimenters for negligible reasons such as playing badly, buying different items than normal, warding in spots that were not suggested, not picking the specific hero suggested, etc.

16 Experimenters received communication bans for 24 hours

10 Experimenters received communication bans for 48 hours

7 Experimenters received communication bans for 7 days

68 Reports were filed by Experimenters

12 Players were MUTED in matches following a report made against them by an Experimenter in a previous match (while in the previous match were not muted)

Every Experimenter experienced at least one communication ban except the one that started in Low-Priority (surprising!!)

An added note, my group of friends has chosen to refrain from purchasing anything from valve until you do fix the current system, they seem to agree with me that in it's current state it only causes more damage to the community than good.

We understand that your profits will not hurt from 34 people not buying, but only hope that our voice adds to others that are very unhappy with you current system of in game "justice" that is creating more silence and fear than communication and teamwork.

Please fix it soon! We are more than confident you can make the system in to something that can improve the community while dealing out punishments that are conductive to a functioning, healthy community that is not terrified of each other.

34 players is far too small of a sample size to be statistically relevant, its the experiences of a mere .00001 percent of players

Also, 34000 players are whats muted at any given time, not muted each week, people cycle in and out of the muted pool. If 34000 people are getting sick, and then all cured within a week its not an epidemic.. THATS THE COMMON COLD. (though personally, I think it seems a bit too easy to get thrown back into the pool after the 7 day bans, it needs some sort of decay mechanic) '

Your numbers suck, all these threads purporting themselves to be about data and statistics all suck because the numbers are FAR FAR FAR too small to be relevant, you need TENS OF THOUSANDS of participants, not a few dozen!

34 players is far too small of a sample size to be statistically relevant, its the experiences of a mere .00001 percent of players

You clearly do not understand statics... as long as the sample is relatively unbiased then the sample size is more or less irrelevant. Most statistic trials (drugs, election polls, business analysis, etc) use a sample size of around n=30. n=34 players is actually a pretty good number to use, not sure if this was intentional by TC or not but either way it is a great number to choose.

If you don't believe me do a quick google search on "n=30 sample size" and you will find numerous explanations as to why any sample size greater than this is generally of little use. There are rare occasions (high volatility within the population) that can make having a large sample useful, but for TC's task it is not necessary.

All I have gathered from your argument of that 34 people is too small to sample, is that you really do not wish to acknowledge how many different people you can potentially run in to while que'ing over 2 weeks. Or that how many people are playing on an exceptionally casual level.

As of right now, the system needs cold medicine & doctors for check-ups because the common cold can be nasty as fk. Not to mention the common cold tends to be highly infectious and generally annoying, exactly what this system is showing itself to be (when it comes to abuse that is)

White knight wont be happy unless someone gathers 10k+ people together to make a graph for him to show how broken the system is.

*edit* I was completely oblivious to the number 34, my next experiment is actually 41 participants. Had I known this earlier I would have kept it 34 for the sake of the spreadsheets that are being drawn up. Maybe I will cut it back down to 34 or somewhere in the 30's.

Last edited by Lynxvarn; 06-23-2013 at 11:33 AM.
Reason: the number 34 stuff

*edit* I was completely oblivious to the number 34, my next experiment is actually 41 participants. Had I known this earlier I would have kept it 34 for the sake of the spreadsheets that are being drawn up. Maybe I will cut it back down to 34 or somewhere in the 30's.

I would just use as many people as you can get. I was simply proving that 34 is not too small of a number to relevant. As always, the larger the sample the better. The important thing is trying to keep the sample unbiased...

I would just use as many people as you can get. I was simply proving that 34 is not too small of a number to relevant. As always, the larger the sample the better. The important thing is trying to keep the sample unbiased...

We don't know that our population has normal distribution, n=30 assumes normal distribution. We don't know that. In fact I think our population may be very skewed one way, which means you need a MUCH larger sample size

because our information on the population is low, assuming 30> is enough is silly, we probably need several thousand to get any data thats meaningful, (also, the sample is biased, since "An added note, my group of friends has chosen to refrain from purchasing anything from valve until you do fix the current system, they seem to agree with me that in it's current state it only causes more damage to the community than good."), so still useless data