Inside Bernie's Penthouse

Madoff's sons Mark and Andrew, his brother Peter and niece Shana will be sued for negligence and breach of fiduciary duty, according to the report.

The action, in civil court, will have no bearing on any criminal prosecution of the Madoff family, which has so far seemed unlikely. In an agreement with prosecutors, Madoffs wife was able to give over all of her riches save $2.5 million allegedly not tied to the fraud.

"Whether or not they have a criminal problem, we will pursue them as far as we can pursue them and if that leads to bankrupting them, then that's what will happen," Picard told "60 Minutes."

Mark, Andrew and Peter took in over $80 million in compensation between 2001 and 2008 and withdrew over $51 million from the investment accounts they had with Bernie. 60 Minutes reports that they are also seeking an extra $90 million in deferred compensation from the now-bankrupt company.

Bernard Madoff & His Victims

"If you were those sons, and you knew what you knew today about where all that money came from wouldn't you be embarrassed to keep that money?" asked Picard's chief counsel, David Sheehan, on the program. "They should give it all back and if they don't give it all back, I think we have an obligation to go get it."

The 71-year-old Madoff is serving a 150-year prison sentence after he revealed a multi-decade fraud last December that cost thousands of investors millions of dollars.

Picard told the CBS news magazine that well over $150 million was either loaned, paid or transferred to the four family members.