vendredi 10 mars 2017

This is a direct replication of my previous experiment on the effects of giving a 50 silver incentive to increase answer rates to a in-game survey. The methods were the same as those previously used [1], with the exception that I whispered players from my /who list in reverse order this time. The players from the bottom of the /who list were first whispered the incentive question and only then were the 25 players in the upper part of the /who list whispered the question without incentives. The experiment was done on Friday 10th of March 2017, 17:17 Server Time.

This change was done because I hypothesized that the order could affect my results and reversing it in the replication would end up balancing the first experiment. For example, there could be more players willing to answer in zones beginning with different alphabetical letters. Should I try this experiment again, I would instead encourage whispering players in a different manner, like this : player 2 on the list, player 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24. And then giving out the second intervention to players 1, 3, 5, etc.

[Results] A total of 44 players were whispered, 24 in the intervention group and 20 in the control group. Players in the intervention group had a mean level of 44.4 and players
in the control group a mean level of 44. A total of five (5) players were
offline when whispered (1 in the intervention group and 4 in the control group) and a total of four (4) players were away from
keyboard (AFK) when whispered (2 in the intervention group and 2 in the control group).

One (1) player in the control group didn't talk english and couldn't participate. Two (2) players, one in each group said I had already whispered them and decline to participate twice. One (1) player in the incentive group didn't want to get the reward twice when I whispered him on his other character, and one (1) player in the intervention group said money wasn't needed "I'm happy to help, No money neccessary, :)".

9 players answered in the intervention group (37.5%) (the player answering twice wasn't counted) and 8 in the control group (33.3%) (the player that didn't speak english wasn't taken into account, neither was the one answering twice).

Relative risks can be calculated by blindly (I know, I know...) using Medcalc [3].
Incentives were associated with a 17% (RR 1.17, 95% CI 0.52 to 2.63) higher chance of getting an answer, although the confidence interval
crossed 1, which means there could be no real difference. The absolute
difference was +4.2% (37.5%-33.3%).

When results from the previous experiment were combined with those new ones* (this means I assume they are similar, which could be wrong), the incentives were associated with a 7% (RR 0.93, 95% CI 0.54 to 1.63) lower chance to get an answer, but again, the results could be due to chance. The absolute difference was -4% (41%-45%).

The reported latencies (ms) were as follows :

EU : 99ms, 53ms, 45ms, 167ms, 47ms, 73ms, 50-100ms, 15-30ms

US : 170ms, 166ms, 100-175ms,

RU : 50-60ms

Unspecified location : 18ms, 195ms, 16-31ms

One (1) player couldn't see his latency due to addons "I cant see latency, sorry" and one (1) gave a narrative answer "hi, today I have experienced some heavy lag spikes, but since tha made the network changes, latency have been good overall" and players added details "its usually good, idk what the average latency is or whatever", "there is always the weird server lag especially if you encounter ppl from NA as European".

I ended up giving out a total of 4 gold (8x50 silver).

Perhaps of interest is that no single player asked me if and where the
results could be seen or what I wanted to do with them. The "further details" message was never sent to
anyone.

Some players seemed particularly happy to participate and friendly "I'm happy to help", "You are doing gods work :)", "no problem thanks you too", "Greetings sir, ofc i can help out with that". I did not get negative comments about the amount being too low.

[Harms] I had to apologize twice for sending the same whispers to characters from the same players and none complained of this being a problem "hi i already said around 60 on another char, no need to send again :p, no problemo", "I have already answered on another character and answering twice would
provide lackluster results, right?, No worries, You are doing gods work
:)".

[Conclusions] The overall results suggest that a 50 silver incentive is associated with a lower chance of getting an answer when players are whispered to participate in an online survey. To get a definitive answer I would need a much larger sample, especially considering how the effect, if any, appears to be small.

In the very best possible scenario, with the maximum relative increase of 63% (RR 1.63), and a baseline answer rate in control groups of 34% (33.3% to 34.8% in my experiments), 50 silver incentives would lead to a 55% answer rate (0.34*1.63). This is a 21% absolute difference, which means I would have to pay about 5 players (5x50 silver = 2 gold and 50 silver) to get one more answer. And in the worst case scenario, giving the 50 silver incentive would lower the amount of players answering. I don't think that 50 silver incentives is worth it for surveys about latency. Perhaps results would be different with another kind of survey or question, or wording.

Latency appeared to be stable between experiments, even if the hour of the day and the day of the week weren't the same. I found players to be quite friendly and patient in their answers.

Acknowledgments

I (Tekai) used to play on the Kronos I server as a level 60 warlock for
months in a progression raiding guild, I wrote extensively about it,
contributed to bugreports, quality improvements, discussions and have
published the (now completed) Kronos WoW Comics (humoristic drawings
inspired from the server) on the official Project Kronos Facebook page. I
don't have any financial interest in this survey and its results and I
am not financed in any way by the Kronos team or anyone else.

This is mostly an exercise of curiosity and for fun (yes my fun is
wierd), please don't take it as a scientific study, but go ahead and do
refer to it if you find it interesting.

17:17:40 [W From] <49:_>: 99ms in EU (_)
17:18:28 [W From] <57:_>: 53 ms in EU.
17:18:33 [W From] <22:_>: dont know how to check, 45, EU
17:20:14 [W From] <17:_>: hi i already said around 60 on another char, no need to send again :p, no problemo
17:20:10 [W From] <60:_>: USE 170 ms on avg.
17:20:35 [W From] <17:_>: my average ms is 166, I live in the US on the East Coast.
17:22:09 [W From] <15:_>: I'm happy to help, No money neccessary, :), 167 UK/EU
17:23:37 [W From] <60:_>: I'm EU (_) and have 47 MS
17:24:38 [W From] <60:_>: 18ms atm
17:24:23 [W From] <60:_>: idk what to tell u its usually good, idk what the average latency is or whatever, ya i knnow mine 195 right now

Control group, 20 players whispered , 2 AFK, 4 missing

17:26:43 [W From] <60:_>: I cant see latency, sorry
17:28:18 [W From] <60:_>: 100-175ms, usa based
17:30:05 [W From] <60:_>: I have already answered on another character and answering twice would provide lackluster results, right?, No worries, You are doing gods work :)
17:30:34 [W From] <60:_>: hi, today I have experienced some heavy lag spikes, but since tha made the network changes, latency have been good overall
17:30:59 [W From] <60:_>: hey, im from Russia and i have 50 ms atm. Usually its around 60ms
17:33:12 [W From] <2:_>: EU and right now its 73, no problem thanks you too
17:33:28 [W From] <60:_>: all from 50-100ms, but there is always the weird server lag especially if you encounter ppl from NA as European
17:33:48 [W From] <42:_>: 16-31ms
17:35:07 [W From] <8:_>: Greetings sir, ofc i can help out with that, Im in _ and my latency varies from 15ms-30ms, same to you!
17:33:32 [W From] <60:_>: sry i dont speak english

*For a total of 16 answers in the incentive group for 39 whispers and 19 answers in the control group for 42 answers.