"I knew," the Brothers & Sisters star, 30, told PEOPLE at Saturday's World Poker Tour Invitational, "and when you know, you know."

But knowing and doing are two different things, and asking the woman he loved to spend the rest of her life with him wasn't something that the actor was going to take lightly.

"I was terrified," a relieved Annable now says with a laugh, "and she could tell."

It's generally accepted that when you pop the question, you don't get a do-over, so Annable approached the task with a mixture of
romance and strategy.

First came the ring, and when Yustman talked, Annable listened. "She only dropped a hint once," he says, describing his blueprint for success. "I wrote it down – the type, the style. I just pulled out my phone and was like, 'Errr, I'm texting my ... agent.' I put it down so that I would always remember."

When it comes to all things ring-related, know when to call in some backup. "I was so stressed going in there," Annable says of his many trips to jeweler Martin Katz. "The woman was like, 'Please, just buy it already!' I was so nervous I brought my mom in."

And don't think that location, location, location applies only to real estate. Annable popped the question in Santa Barbara, Calif., over the Valentine's Day weekend at the spot that they "fell in love."

With all of the planning involved, Annable felt sure that he had the element of surprise on his side – he just didn't realize that it might work against him.