"Why would he go through all this effort to have this fantastic couple of days, to have our most intimate dearest friends come join us, for us to be reluctant?" the former Olympic swimmer, 33, told Matt Lauer on Wednesday's Today show as her husband, 53, sat next to her. "It sounded a bit delirious."

"So, there was no 'runaway bride'?" pressed Lauer, who also mentioned the rumor that Monaco palace guards followed her to the Nice airport and took away her passport.

"No. Not at all," she said. Her husband explained the origins of such stories by saying, "I think there were ... people that were envious or did not like the fact that we were finally … together and marrying."

Tuesday night, the prince and princess were on the town in New York City to honor Albert's late mother, Grace Kelly, at the 29th annual Princess Grace Awards, which support emerging talent in the arts.

With Princess Charlene (in Dior) at his side, the Prince told PEOPLE about all their travels. "We haven't gone completely around the world," he said, "but we've been to a lot of different places."