It’s a time for family, but in Josh Duggar‘s case, he may very well spend Christmas far away from his.

The eldest Duggar sibling checked into rehab this past August, after confessing he had been unfaithful to wife Anna and had become addicted to pornography. Now, a source close to the family says it would be a surprise if he returns home for the holidays.

“I don’t think he’s going to be back for Christmas,” says the source. “I would be surprised. There would be a huge elephant in the room.”

Josh’s most recent scandal has taken center stage on his sister’s new TLC special Jill & Jessa: Counting On, which airs Sundays on the network. On the show, a host of his 18 siblings weighed in on how much the shocking revelations hurt them and the family.

But the source says there are at least four little Duggars eagerly awaiting his return.

“It’s so hard when it comes to the kids,” says the source of Josh and Anna’s four children: Marcus, Mackenzie, Michael and Meredith Grace. “They miss their dad. They’re like ‘Where’s daddy?’ They’ve said it before.”

“She said, ‘It was the best three days of my life,’ says the source. The young mother was similarly positive on the most recent episode of Counting On, saying in an on-camera interview that her “prayer and hope is for our marriage to be restored.”

One thing is certain – the family is focusing on forgiveness this season. Says the source, “[Josh’s dad] Jim Bob keeps saying this is just another new level of forgiveness.”