It’s not every day the most decorated Olympian of all time strolls into your living room, so Lori Loughlin feels pretty lucky.

PEOPLE Now recently caught up with the Fuller House star, who revealed the backstory behind her epic Instagram pic with Michael Phelps back in August.

“We didn’t know he was coming over,” said Loughlin, 52. “What happened was my husband was out playing golf with some buddies, and one of the guys is Michael’s good friend and then they came back to our house and they were all going to go to dinner.”

According to Loughlin, the group stopped by her house to “have a cocktail,” and the next thing she knew her husband Mossimo Giannulli was calling her over.

“And I walk into our living room area and Michael Phelps is standing there!” she said. “I was like: What?! But he is so incredibly sweet.”

The best part? The chance meeting was right after the Rio Olympics, where Phelps, 31, took home six medals in total (five gold and one silver), making him the most decorated athlete in Olympic history with 23 gold medals and 28 overall.

“I said: ‘It’s incredible what you’ve done,’ and he was so humble,” recalled Loughlin. “He said: ‘I don’t know if I’ll ever be able to wrap my mind around this.'”

Loughlin said Phelps showed her pictures of his medals — and his adorable son Boomer, 6 months, whom he shares with wife Nicole, 31.

“I asked him about Boomer and he showed me a lot of pictures. He’s such a sweet little boy,” gushed the star. “But you can just tell, he’s so happy, he’s so in love, and I think he’s in a really good place. He’s just a really sweet guy. That was my only encounter with him, but he’s quite impressive.”