Saturday, December 20, 2008

At about 4:15 a.m. Eastern time this morning, a most curious hand of limit hold’em (blinds $200/$400; limits $400/$800) appears to have taken place on UltimateBet between a player named “DOUBLEBALLER” and “PHILHELLMUTH,” the latter being, of course, the 11-time WSOP bracelet winner and high-profile spokesman for the online poker site.

In the hand, Hellmuth was the button/small blind and DOUBLEBALLER the big blind. The players capped the betting preflop. The flop came . DOUBLEBALLER checked, Hellmuth bet $200, DOUBLEBALLER check-raised to $400, and Hellmuth called. The turn was the . This time DOUBLEBALLER bet out ($400), Hellmuth raised, DOUBLEBALLER three-bet, and Hellmuth called. The river was the . DOUBLEBALLER bet, and Hellmuth just called. Total pot ended up being $5,600 (minus the one buck rake).

DOUBLEBALLER showed for three of a kind. Hellmuth mucked. It turns out he had .

So what makes the hand curious? The pot was awarded to Hellmuth. (Click on the hand history above for bigger picture.)

An hour or so after the 2+2 thread started, DOUBLEBALLER hopped on to confirm what had happened, and to relate that he was on the phone with UB support requesting he be reimbursed the money he’d lost. Apparently, support acknowledged to him that a mistake had been made and indeed credited DOUBLEBALLER’s account. (No word in the thread thus far as to whether the amount was also deducted from Hellmuth’s.)

DOUBLEBALLER reports that at some point during the hand a pop-up strangely appeared telling him he had missed three blinds in a row, even though he had not. That message appears to have been somehow related to whatever glitch caused the software to ship the pot incorrectly.

The 2+2 thread includes some of the players’ chat (revealing Hellmuth to be somewhat oblivious of what had happened). There are also various posts independently confirming that the hand indeed took place as described, as well as at least one poster sharing a response from UB support after he’d emailed them asking about the incident. In the response, UB support explains that they are aware of what happened, and that “this situation was due to a glitch in the software, however; it has been already corrected. As you know after CEREUS launch we have been experiencing some glitches in our software but our programmers have been working very hard to correct these issues.”

Some in the thread continue to doubt the validity of the story. Others are wondering how exactly such a mistake or “glitch” could possibly occur. And many are particularly intrigued by the fact that the glitch -- a seemingly unlikely event -- would happen with player PHILHELLMUTH at the table.

This sort of thing happens a lot when you have a system, like a poker game server, that has to handle multiple tasks doing essentially the same thing at the same time.

Certain precautions have to be taken in the software to maintain the integrity of the data associated with each task and to make sure each task doesn't accidentally use or alter the data belonging to another task executing the same bit of code. This can get complicated, and it's easy for programmers to screw it up (although as with all things, you get what you pay for - really first-rate programmers with a good grasp of multithreading and multitasking issues in realtime systems generally don't have a lot of bugs of this type).

The other way this can happen is that the code releases memory allocated for a bit of data before it's actually done using that data. That chunk of memory then gets allocated to a different task, which writes to it. But the original task is still looking at that memory, only now it's looking at some other task's data. To make things worse, that new data might be aligned on a different address boundary, so the original task gets really confused.

In short, there are lots of entirely plausible, mundane explanations of why this can happen. That's not to excuse it, and if I were the head programmer on some of these sites (not just UB), heads would be rolling if this crap got put into production. But the sort of bugs we're talking about are pretty routine among careless, lazy or stressed-out programmers.

I just want to add that the nature of these things makes them intermittent, unpredictable, and hard to reproduce. Bugs like this can sometimes go for months or even years without being detected because the weird combination of events that must happen to trigger them sometimes have a very low probability of occuring. I have first-hand knowledge of a system that had such a bug for seven years before it finally caused a system crash.