Bernie Madoff expected to be caught earlier, according to a lawyer representing his victims.

"He thought in the year 2000, 2001, to use the colloquial phrase, the gig was up," said Joseph Cotchett, who talked with Madoff for four hours Tuesday at the Butner Federal Correctional Institution near Research Triangle Park, North Carolina.

"In 2005, the SEC [Securities and Exchange Commission] sat down with him for a full day and grilled him, and he told me that when he left that meeting he thought he would be closed down."

Cotchett told CNN's Campbell Brown that Madoff told him if the SEC had discovered his fraudulent scheme then that billions of dollars would have been recovered.

"Had he been closed down, $20 billion would have been saved of investors' money," Cotchett said he was told.

Madoff was convicted of operating a Ponzi scheme and defrauding thousands of investors. He pleaded guilty and was sentenced to 150 years in prison this year. Prosecutors have said it was the largest investor fraud committed by a single person, totaling billions in losses to investors.

As for Madoff's adjustment from living as a fat cat on New York's Upper East Side to living behind bars, Cotchett said Madoff is doing well.

Don't Miss

"He is a bit of a celebrity. As a matter of fact, people are asking him for autographs, which he offered he's not giving," Cotchett said. "He exercises. He walks around the track. He spends a lot of time reading. And he's got a job. He's in the sign factory, if you want to call it that, where he paints signs and clerks can make sure that signs get painted."

Cotchett described Madoff's response after being asked whether he felt remorseful for his actions.

"There was a silence for a moment. He leaned over in his chair and he said, 'Tell them how sorry I am.' Now, those are mere words. Whether or not in this guy's heart he's sorry, one thing is for sure: he's not happy in the circumstances he now finds himself. But sorry is immaterial, because sorry is not going to get any money back for these victims."