In his first prison interview, a buff-looking Bernie Madoff said he couldn't believe he got away with his massive Ponzi scheme for so long.

"There were several times that I met with the SEC and thought 'they got me,'" Madoff told Joseph Cotchett, a San Francisco lawyer threatening to sue his wife, sons and brother on behalf of a group of victims.

Cotchett said he and his partner, Nancy Fineman, met with Madoff for four and a half hours Tuesday afternoon at the federal prison in Butner, NC, where Madoff is serving his 150-year sentence.

"He looked pretty good and seems to be working out," said Cotchett. "He looked a lot better than he has in some months since I've seen photographs of him."

Cotchett said Madoff was "very articulate, very direct" and did not appear to hold back anything. "He talked about how he pulled it off, how many years he got away with it," the lawyer said.

"I was surprised at how candid he was," Cotchett told ABCNews.com after the session, the first time Madoff has talked with outside lawyers. Madoff refused to cooperate with the FBI after his initial, largely untruthful confession last December.

Cotchett said Madoff "did not dodge" any of the questions he asked and that Madoff's lawyer did not object to any of the questions.

"He obviously wanted to speak with us because in his opinion, certain members of his family knew nothing about it, had no involvement of it," said Cotchett who was able to arrange the unusual session after threatening to sue Madoff's wife Ruth.

"He cares about Ruth," said Cotchett, "but he doesn't give a ---- about his two sons, Mark and Andrew." The sons have not spoken with their father or mother since Madoff's arrest on December 11. They say there were unaware of the fraud scheme until he confessed to them as his money was running out and it appeared the crime would be exposed.

Cotchett said he did not yet know if he would name Ruth or the sons in the lawsuit, but that he was almost certain to name Madoff's brother, Peter, who served as the firm's chief compliance officer.

Peter Madoff has declined to comment but others with knowledge of the case say he maintains he did not know of the scam until two days before his brother's arrest.