Amid a storm of controversy and police investigation, a California teacher who left his family and job to be with a teenage former student said he followed his heart and that the couple waited until he resigned before they "took it to the next level."

"I know my family is probably disappointed in me," James Hooker, 41, said. "They're probably confused in what we're doing. I'm not sure how to explain it to them. ... I am just following my heart and this is the direction I'm going in."

At issue, however, is when the relationship between Hooker and 18-year-old girlfriend Jordan Powers truly began, and whether Hooker broke any laws.

The former teacher at Modesto's James Enochs High School maintains that he did nothing illegal, and that people questioning the couple's relationship, including Powers' mother, who wants Hooker arrested, are ignoring the details of their relationship's timeline.

"They don't care about the rest of the story. They don't care about the rest of the detail," Hooker said. "Once I ... resigned and ended my assignment ... that's when we took it to the next level."

Powers, who has stopped attending classes at her high school but is still planning to graduate on time by doing independent study, said she's in love with her former teacher, who she met over three years ago.

"Yes, I do love him," Powers said."He's my best friend. I mean, he's more than just a lover."

As soon as the couple went public with their love on "Good Morning America" Thursday, criticism of their romance went viral, and raised eyebrows across the country.

Hooker and Powers met when she was a freshman and he was her business class teacher. They say their relationship evolved over time, and only became physical after Powers turned 18 in September.

"We don't expect people to understand," Powers said.

Hooker seems to agree, saying that a relationship like theirs will generally be misunderstood.

"It's just difficult to hear things that aren't true, and to be perceived as wrong no matter what our case is," Hooker said.

Tammie Powers, Jordan Powers' mother, is not convinced by Hooker's claims, and has taken her outrage to Facebook, along with her contention that she found phone records showing long, late-night phone calls and more than 8,000 text messages between the couple.

"He shouldn't be with kids. I'd like to see him arrested. I'd like to get information from when she was a minor and have him arrested," Tammie Powers said.

In California, where it is illegal for anyone younger than 18 to have sex with an adult, police are now investigating.

"The fact that there were these 8,000 messages, there was an ongoing relationship that was extracurricular, that doesn't pass the smell test," criminal attorney David Cohen said.

As for his relationship with his family, Hooker said he plans to file for divorce, but he hopes he can salvage a relationship with his three children, including a 17-year-old daughter who attends the same high school where he taught and his girlfriend plans to graduate from this spring.