Jesse James's Employees Rally Around Their Boss

With his familiar blue baseball cap shading his eyes, Jesse James keeps on working – it's just about the only thing he knows to do right now. Running West Coast Choppers in Long Beach, Calif., seems to be his only solace these days while his marriage is in crisis.

But as much as he is still supported by his workers, including two female employees who spoke to PEOPLE, there's no denying the fallout from allegations that their boss cheated on his Oscar-winning wife, Sandra Bullock.

James, this worker says, is "keeping to himself right now," putting on a brave face when his children are around. "He's showing them that he's fine, but I really know he's not," she says. "You can just tell by the way he sits in his office. He has a blank look."

Every day now, it seems, brings another wave of bad news. It started when Michelle McGee, a San Diego tattoo model/stripper, claimed last week that she had a months-long affair with James. Then came allegations from another tattooed stripper, Melissa Smith, of a long affair, followed by allegations from an L.A. photographer that she was intimate with the motorcycle mogul. On Thursday, an unidentified woman, who has hired celebrity lawyer Gloria Allred, claimed she had an affair with James that only ended after the scandal broke.

For James's employees it's almost too much to process – and much more than they're willing to believe. They prefer to speak of the James they know.

"Jesse is like family," says the employee. "He's just a righteous guy. If I have a personal problem, he'll sit and talk to me. Not once have I felt uncomfortable with him ... He's just a sweet guy. Hearing all this stuff about him is just sad. I don't believe any of it."

Another female employee tells PEOPLE: "He's a great guy. Working for him has been a great experience. He's always been very respectful ... He’s got a good heart. It's been nothing but a great experience."

This employee lists all the good James has done, from being a devoted father to his charitable work in the Long Beach community. "The way things have been portrayed is extremely unfair and unjust," the worker says.

And every day, she says, she sees the strain on him.

"He's just trying to keep working hard," the employee says. "He's trying to concentrate on work and his kids. That's what's helping him get through all of this."