The PL/SQL Challenge (www.plsqlchallenge.com) offers a daily quiz on the PL/SQL language, through which thousands of Oracle technologists demonstrate and deepen their knowledge of PL/SQL. This blog contains posts by the PL/SQL Challenge founder, Steven Feuerstein, as well as comments from players.

05 October 2013

Sad Day for the PL/SQL Challenge: Cheaters Removed

Since the early days of the PL/SQL Challenge, we have been aware of the possibility of cheating by players. In response, we developed a set of tools to analyze player data to identify patterns of answer submissions that could reflect one or another form of cheating (such as a single person having multiple accounts, or multiple people colluding to achieve high scores).

We have in the past applied these tools to identify players who answer too quickly, and made adjustments to their scores accordingly. When we apply the adjustment, we notify the players that:

It is certainly quite possible that a person answering so quickly (and correctly) is cheating by taking the quiz under a different account, figuring out the correct answers, and then submitting the answer under their "real" account. It is also possible that they knew that subject very well or simply guessed. Rather than automatically reject such answers, we analyze player patterns over a longer period of time, usually no less than two weeks. We then make adjustments to the timings and ranks of those players who answer, cumulatively, in so short an amount of time that there is no way to distinguish their pattern of play from someone who would be cheating.

Today, it is my distinct non-privilege to announce that I have removed from the website three players who I believe have colluded over many quarters to take a whole bunch of the top prizes in the quarterly PL/SQL Championships. The players are (with their championship history under their names):

As you can see, two of our consistent champions dropped dramatically in ranking when in the Q2 2013 championship everyone played at the same time.

After I notified all three players of my determination and asked for a response, all three "people" immediately stopped playing. Only Frank even replied to my request and he protested his innocence. Unfortunately, it was not a convincing protest. The bottom line is that there is no way to distinguish their pattern of play from someone who would be cheating.

Frank and Konrad cumulatively won $2800 in prizes on the PL/SQL Challenge site. This is, of course, a source of extreme frustration for me....but what can you do?

Well, here's what we can do: all rankings have been recalculated after these players' removal. We therefore have in many cases new winners for the championships. I am sorry that we will not be able to issue prizes to these new winners, but at least you will be able to print off and display your certificate of achievement - and these new achievements become a part of your profile and history on the PL/SQL Challenge website.

And - do I really need to say this? - let this be a warning to other players who are cheating or contemplating such: we will continue to enhance and apply our tools to root our cheaters, remove them from the site, and announce their crimes to the world.

We may also make our tools available on the website itself, so that any players can run the analysis and see if they can uncover cases of potential cheating to report to us.

32 comments:

I'm also from GERMANY. Frank was a kind of model for me. Too bad...I can't understand such a behaviour. Some, not playing collegues, responsed me in the past to do the same. But that's not the sense of this quiz for me. I use it or my further education and I LOVE it AND I'm very, very thankfully to Steven and the other developers of this quiz!st_guitar

I wonder about my ranking in the cheaters' top. This championship was by far the most unpleasant for me - after a 9-hour workday I was not able to read, remember and analyze the complicated pieces of code. I also dropped quite a lot when compared to previous championships. I have never won a monetary prize in these events, but I think that my "behaviour" qualifies as cheater by these standards?

I came second in one of those. This news doesn't really cheer me up though. It takes significant effort to play the contest competitively on a daily basis. You really can't afford to miss even a day. I guess some people got a little too obsessive about it.

slimkask - it wasn't the performance on one championship that led me to this action. It was a long pattern, and relationships between the activity of several players. So, no, your single rankings drop does NOT raise a red flag.

Matthias - if it's any consolation, I don't think there were three Germans. Maybe just one one or two. :-(

given, that, as the above linkshttps://www.xing.com/profiles/Frank_Schrader17https://www.xing.com/profiles/Dieter_Kowalskihttps://www.xing.com/profiles/Konrad_Schmidt19show, Frank Schrader works in Braunschweig ( Brunswick ), Dieter Kowalski works in Wolfsburg, and Konrad Schmidt works in Halle/Saale,and due to their names it seems quite likely they all three are Germans.For those not familiar with German geography, http://tinyurl.com/oo68xwqshows the location of these three towns in Northern Germany

Sorry, should have been more clear. I am not certain that there are actually three distinct living and breathing humans. Maybe just two, maybe in actuality just one. One person could pretend to be two people playing the championship at the same time. And while those locations are different, the IP addresses when quizzes were played tended to show something else.

It's kind of upsetting. I hope Frank was a cheater, because I did not see anything wrong with his performance, some other players that have vanished after the announcement of the "No Cheating policy" were obviously cheaters.It make me feel vulnerable as well as when I am going on holidays I don’t have anywhere to test my answers so I answer quickly just by knowing enough of the subject or guessing. Does it mean I might be qualified as a cheater?

a sad day for the PL/SQL challenge indeed. I'm kind of stunned - I'd never have thought that cheating on this scale was possible (since obviously, cheating in the playoffs requires collaboration of at least two playoff participants). Even now, I'm hesitant to believe this, although I guess Steven must have conclusive proof.

Regarding the possibility of making the tools for checking available: I'm of two minds about this. Although I believe in open-sourcing algorithms and software (more eyes=fewer bugs), you would also have to publish all raw results from the daily challenge. This might allow potential cheaters to spot bugs in the detection algorithm or to "fine-tune" their cheating to stay below the radar.

Anna, as I stated earlier, it is not a matter of just a single answer or performance on a single championship. It is a matter of analyzing a long-term pattern of behavior that involves at least one other player, and sometimes more. So, no, if you guess and answer quickly on a quiz, you will not be flagged as a cheater. If you start answering really quickly all the time with 100% correctness, and then in the playoff you blast through all obstacles to a 100% score in 5 minutes, well, then, yes, you would come under suspicion.

too bad for them! It's their loss. My main prize is to learn with all of you and challenge myself everyday. And what a journey has been so far. As an active player, i don't give a damn to where their prizes are. Just keep on forward!

Here's a suggestion: i know you want plsqlchallenge to be free but how about charging a symbolic amount (1$/year ?). Those kind of players would stay away, or am i being too naive?

cheaters are really bad and shows no real valuation for creating the mass of quizzes and administration of the website.

I think, charging a symbolic amount could some of them get out of the challenge, but this could victimize mass of players, who are not cheating.

My personal motivation for playing is to learn about new or unknown features, so for me prizes are not really necessary. The prizes are a nice gift for rewarding good knowledge, but they can be a wrong incentive for cheaters.

Steve, I don't know if I remember correct: was not planned a change in the prices to reduce the costs of website? Probably, the prices could be reduced to maximal 10$ for the best or better, no distribution of any prices.Without monetary prices it is only played for honour, which is sufficient for me. This will not stop all cheaters, but I think, most of them will not invest much effort without monetary incentive.

Bruno, I like your idea and have entertained it in the past, in various forms, but I tend to think that MANY players would stop playing to avoid giving their credit card info for something that is free, and something that is not really necessary for them.

In your poll on payments it's not very clear per what we are voting to pay. Is it per Quiz? Per Week? Per Month? Per Quarter? etc or it's once off? Because my answer is different depending on the above.

I fully agree with Gary Myers's post above, in that it takes a high degree of commitmentand not a small effort to play daily, along several years, and sacrificing a lot for not missing any quiz.

My personal feeling was always that the Championship Playoff ranking in the Top3 was usually celebrated with "a little bit too much drums", and this even without any relation to the money prize that accompanied it up no now.

If we look back at the first few Playoffs played, then along all these years the Playoffs have becomemore and more difficult with the time ... so, if ranking on a low place in a Playoff or even in several ones would always mean cheating, then most of the Playoff participants, including probably ALL the winners up to now could easily be categorizable as cheaters, exactly as Siim Kask said.

When I said "too much drums", what I wanted to say was that, again, in my opinion,the Playoff is maybe the kind of competition which reflects in the lowest degree the long-time commitment and devotion of a player to the PL/SQL Challenge ... at the other end, in my opinion, stands the "Lifetime ranking", which goes almost unnoticed when compared to the Playoff ...

Maybe this "too shining" color of the Playoff became kind of "naturally attractive" for cheaters ...

I am confident, however that, as Steven said, the algorithms implemented are strong and reliable enough for not producing "false positives" ... this would be too sad for any Playoff participant that just happens to rank low, something that, again, may happen at any time, to anyone of us .

I really appreciate your effort in removing cheaters from this forum and I assume that all genuine players play ONLY to test and enhance their knowledge. So probably you can consider doing away with cash (Amazon card) prizes and award only Books. I am sure this will never be a motivation for a cheater to play the quizzes and at the same time this sort of prize could be the most precious one for the genuine learners. Probably you may ask the winner to select a book from a list of books (including books on other technologies like Java, Unix) just in case he/she already won an Oracle book and now may want to learn some other technology as well. In this way we can ensure the prizes are actually helping winners to learn even more.

You can not only select books at O'Reilly that are on the page that is mentioned in the "winner" emails. I have often written an email to O'Reilly because the book I wanted to recveive wasn't in the list of possible books. In the section titled "Before you request your Ebook please be sure to do the following:" there is one important item in the list:

"Fill out the form below to submit your request. If the book you are interested in is not listed below please email orders@oreilly.com."

So just forward your winning message with a request for the book you want to recveive to "orders@oreilly.com".

And you never got any complaint from them? They just got you your book? OReilly is so nice....but I am not sure if I want to put that in the message to players - because OReilly has NOT told me that officially. Hate to abuse their generosity.

Ah, but I stand corrected on one thing: you are NOT restricted just to Oracle books. We SAID that but it wasn't true - there is a Loooooong list of books to choose from.

I just want to mention that once I wrote to O'Reilly and asked about a book that was not in thelist of books that accompanied the e-mail, but was available while browsing the O'Reilly shop site( as in the "normal" way of shopping on a web site ).

The answer that I received, however, was that "the book I asked about is *not* included in this promotion",so, from here I understood that we have to choose only from the list of books that accompanies thee-mail we receive.

The list of books is indeed very long, though, in comparison with other topics,the number of Oracle-related titles is not very large.

Steven, You deserve a price!As an alternative you can select 10 charity organisations, the winner can choose what organisation gets the money, his price is that he or she is mentioned in the hall of fame with the price and organisation mentioned.Regards,Dik Pater.The Netherlands.