While some people’s unsavory secrets are catching up with them in Masters of Sex episode 4, “Dirty Jobs,” nasty prejudices are highlighting the insecurities of others. At heart these characters are all good people, but a show like this isn’t interested in counting brownie points. So let’s examine what we learned about each character’s secrets and their “dark side” this week instead.

The secret: Virginia and Bill’s affair What we learned: Virginia is feeling guilty about sleeping with her married boss as concerns about her job grow

A heated discussion between Virginia and Bill kicks off the episode; Virginia is upset he resumed the study without her. But the board at Memorial Hospital was “not impressed” with her application as a research assistant, and Bill echoes that sentiment. “You thought it was about the work, but I know the truth,” he says. Which is? Well, on the application she wrote “mistress” on the line asking for her current occupation.

And then Virginia wakes up.

She’s in the hotel room with Bill. He tries to be cutesy—no seriously, he makes endearing comments about the way she sleeps—but the dream has rattled her, and she’s short with him. She’s not only feeling guilt over the repeated hotel rendezvous, she’s unsure about the state of their relationship. She’s also concerned that, without the extra money coming in from the study, she’ll be unable to pay rent. Unfortunately, we learn, she’s still trying to hawk those diet pills to make some extra cash. (Which, you have to admit, led to some highlights of the episode, including the cynical Cal-O-Metric rep who told Virginia to go ahead and blindly follow her dreams of a career in medicine—maybe she’s special and that will actually work out for her.)

But she’s right in worrying, because serial philanderer Langham—who stays at the hotel with his kids when it’s his night with them—spies Virginia and Bill exiting the hotel. And this guy is as loose with his mouth as he is with his pants.

In the lunchroom, Langham tells Virginia he saw her and the doc at the hotel and assumes they’re having an affair. Virginia, quick on her feet, tells him they’ve resumed the study and a hotel is the only place they can get any work (ahem, “work”) done with any amount of seriousness. She wonders if he buys it. He totally doesn’t buy it.

The secret: Bill and Virginia’s affair Bill’s dedication to the study—and Virginia—makes him a bad employee

Bill keeps insisting he wants Virginia to “stay connected to the work,” which, of course, connects directly back to that desperate “I need you” line from the season 1 finale. It’s not just about the work for him—it’s about having her in his life. “Work” for these two, is a synonym for “love.”

Virginia has a hard time believing that’s he’s doing his damnedest to bring her over to Memorial with him. The thing is, he actually is trying his damnedest to convince Greathouse that she’s indispensable to the research. And she is: Without Virginia’s people skills—and the fact that some of the research patients only feel comfortable participating in the study knowing another woman is part of the observation team and it’s not just one sleazy guy—Bill is losing participants.

Still, Greathouse insists he sit in on Bill’s research that evening. Bill tries to discourage his boss from meddling in his work by saying he’ll be looking at men with enlarged prostates that evening—”OLD MEN MASTURBATING!”—he yells, exasperated, at the daft secretary who will be passing along the message. Sometimes Bill’s impatience and short temper are awesome.

Greathouse may honestly be interested in Bill’s work, but he comes across as pervy whenever he talks about the study. He tries to use the “in the interest of medicine” excuse, but Bill knows he’s curious about the study on a much more primal level. Greathouse shows up to Elderly Dudes Jacking Off night anyway, surprised to see an older woman on the exam table. Bill passes it off as a last-minute schedule change; Greathouse makes derogatory remarks as the woman uses Ulysses. Bill again tries to ask if his boss has made any progress in hiring Virginia; Greathouse insists he talked to the board, but “this is a process,” he says, right as the test subject climaxes. (Sex metaphors! Maybe the writing staff is pitching new ideas for the opening credits next season.)

Apparently, Greathouse wants to see more. Bill tries to brush him off yet again, explaining the “transference effect” that could, essentially, cause the untrained observer to develop homosexual tendencies if he’s in close proximity to another male researcher. “Like wrestling?” Greathouse asks. “Something like that,” Bill says, with no luck. Greathouse is smarter than Bill gives him credit for.

Later, Bill tries to calm the nerves of a pretty young female study subject. He enters the observation room and is horrified to see Greathouse and a handful of other male doctors chowing down on Chinese food, getting ready for the “show.” Bill, of course, is livid—another testament to his deep commitment to not only the research, but to his patients as well. Bill pulls Greathouse into the hall and launches into the best speech of the episode (of the season?): “This is a scientific study, not a stag film in a frat house,” he seethes. “You have grossly misjudged me if you think I’m going to allow those baboons in my exam room, slurping chop suey, mocking my work. The terms of my employment—the terms between you and me—are changing now,” he adds. “You will clear my exam room and you will never get near it again. And you will authorize Mrs. Johnson’s hiring starting tomorrow morning.”

Greathouse all but laughs at him. It turns out he never brought up the idea of hiring Virginia to the board: “I’m keeping you from being perceived as a man who thinks with his cock, not his head,” he tells Bill. “Your colleagues are on your side, so relax,” he says, walking back into the exam room. It’s rare that Bill doesn’t get his way in a professional setting… but he ends up having the final say.

Remember last week when Bill mentioned he used to be a fighter? He puts those moves to good use back in the exam room, first shoving an egg roll into one doc’s face, then clocking Greathouse not once, but twice, square in the nose. Of course he’s going to be fired.

The woman study subject, unwitting to what is happening behind the one-way mirror, begins to orgasm. Despite the melee, the men all turn to watch. Bill, disgusted, rushes into the room and covers her with a gown, closing the shade on the peep show.

At home, he doesn’t have a chance to let Libby know he’s going to be fired. Langham has dropped by, and he wants to offer his colleague some advice. (For a second you wonder: Did he tell Libby anything?) The men head to the backyard to smoke cigars; there, with Libby in clear view through the kitchen window, Langham pontificates on the highs and lows of the bachelor life. (Good: all the sex. Bad: He wishes he could be with his kids). Langham used to think: Why can’t I be more like Bill Masters, a successful doctor, a happy family man? But now he knows Bill Masters—the can’t-keep-a-job doctor, the adulterer—is no better than he is. “Whatever you have with Virginia, weigh it against all of this,” Langham warns Bill. “Is it worth it?”

Bill seems to ponder that question the next morning as Libby reams him out for not telling her he has been fired. (She just got off the phone with Mrs. Greathouse, who broke the news; Bill never got around to telling her after his chat with Langham.) But she’s not upset for him; she’s mad at him and fearful for her own cushy livelihood. “Why are you doing this to me?” she asks, a telling line. She scolds him for never being satisfied (which is more true than she realizes) and squandering opportunities because he’s too focused on his research. Doesn’t he think of his wife and child?

“That’s all I think about,” Bill says. “I’ll take care of you, whatever happens. Please, Libby…” and he begins to have trouble breathing, as if he’s having a panic attack.

This is such a contrast to the Bill we’ve seen throughout the entire series. He’s rarely—if ever—shown true love or affection for his wife. But this isn’t a visceral reaction to disappointing Libby, is it? It’s the physical manifestation of his fear that he’s not living up to his duties as a man and that he’s a failure as a husband and a father—just like his dad. Replace Libby with any pretty housewife and his reaction would be the same—unless that woman were Virginia.

We know this because the next time we see Bill, he’s accepting a position at Buell Green Hospital—a black hospital. He “feels comfortable speaking for his partner” that they are very excited to bring their study to this new facility. “Partner”? Virginia is no longer simply his secretary, or even his research assistant. If Virginia is his now his “partner,” then he truly has taken Langham’s words to heart: He’s aware of what he will lose by continuing his work/affair with Virginia, and he is willing to leave it all behind to be with her.

NEXT: The secrets of Lillian, Libby, Betty, and Greathouse are revealed