If the Oscar winner was, indeed, pushing his luck, then he must have been good, a legendary card counter tells PEOPLE exclusively.

“For them to kick him out, he had to have been blatant about card counting,” says Jeff Ma, the inspiration behind the New York Times best-selling book Bringing Down the House, about M.I.T. students who swindled Vegas casinos with their card-counting acumen. (It was later turned into the 2008 movie 21, starring Kevin Spacey.)

“He must have been pretty good at it, too, meaning he was betting the right amount every time,” Ma says speculatively. “For [the Hard Rock] to determine that they no longer want to deal blackjack to him, he must have been doing it pretty well.”

Ma’s theory largely echoes what a source previously said. The insider told PEOPLE that Affleck, 41, was banned from playing blackjack at the Hard Rock Hotel & Casino in Las Vegas because he was simply “too good.”

Ma, a former M.I.T. student who was part of an elite card-counting team that won $5 million in Las Vegas over a seven-year span, is banned from playing blackjack in Las Vegas casinos. Much like he and his teammates did in besting the casinos, Ma assumes those who want to buck the system use “basic math” to count cards and get an advantage.

“Basically, blackjack is the only game in the casino with a memory, meaning what you see impacts what you will see,” he explains to PEOPLE. “In blackjack, it’s been determined that when there are a lot of low cards left in the deck, it’s in the dealer’s favor. When there are a lot of high cards, it’s in the player’s favor. All you’re doing is tracking.”

Should someone decide to keep track, Ma notes, the player is “probably keeping a running total of all cards he’s seen on the table, so you know what high cards and low cards remain. When the odds are in your favor, you increase your bet, and vice versa.”

Contrary to folklore, card counting isn’t illegal, but it is frowned upon by casinos and provides grounds for ejection. And, according to Ma, card counting doesn’t really give players that big of an advantage.

“Depending on the rules of the casino, you can get your odds anywhere between a 1.5 to 2 percent advantage. It depends on how aggressive you’re going to be with your betting,” he said. “You could win a lot with that advantage. Most casinos don’t understand how small of an advantage you have when you’re card counting – they just know you are, and they freak out because they think they’re going to lose to you.”