Located in the sprawling parkland of the estate, “it’s pretty dilapidated now,” veteran royals biographer Ingrid Seward, author of the recent book My Husband and I, tells PEOPLE in this week’s issue. The couple have already put in plans to freshen the landscaping.

Of course, the renovated home will require a nursery in advance of the birth of the couple’s first child this spring.

Although some had pointed to Apartment 1 at Kensington Palace — right next to Prince William and Kate Middleton‘s home base — as a new family residence, “it would need considerable, lengthy refurbishment,” a royal source tells PEOPLE. Windsor, meanwhile, “has more space for children” and “has a special place in their hearts,” the source adds.

Meghan Markle and Prince Harry depart for their evening wedding reception at Frogmore House on May 19.

Long at home in the countryside, “Harry feels very imprisoned at Kensington Palace,” says Seward — and has told friends as much. “He can’t wait to get out.”

Can’t get enough of PEOPLE’s Royals coverage? Sign up for our newsletterto get the latest updates on Kate Middleton, Meghan Markle and more!