Jon Hamm Cannot Be Tricked – Though Mad Men Cast Tries

You wouldn't mess with Don Draper around the office at Sterling Cooper Draper Pryce. And you shouldn't mess with Jon Hamm, the man behind those suits and ties, on the Mad Men set.

But his costars thought about it.

John Slattery, who plays troubled executive Roger Sterling, and Elisabeth Moss, who plays secretary-turned-ad copywriter Peggy Olson on the AMC series – whose season 5 premiere is Sunday – attempted to play a prank on Hamm when he was making his directorial debut for the show's April 1 episode.

"We were going to do this thing when you just mouth the words with no volume, no sound, coming out, and it wasn't even that funny," Slattery, 49, tells PEOPLE about the abandoned idea to trick Hamm. "Elisabeth [Moss] would say that I chickened out, but I think I was just trying to concentrate on what I was doing."

They were not the only ones celebrating April Fools' Day a little early on-set, and Slattery says Vincent Kartheiser, who suits up as young exec Pete Campbell, also tried a hand at tricking Hamm.

"That didn't work either," Slattery says. "Hamm is too smart. He kind of is un-Punkable."

Hamm, after all, had to use those smarts in the directing chair. "It was not my choice to come in and go, 'I'm gonna do my thing. Here's a whole different attitude!' " Hamm, 41, explains. "My job was to not drive that train off the tracks, but keep it going as smoothly as I could."

So was he happy with the final product? "I think I did an okay job," he says. "We'll see."