(CNN) -- "Obviously, first of all, this conversation never took place. ... OK?"

Financier Bernard Madoff is serving a 150-year prison term for defrauding thousands of investors.

So began a phone call in which convicted swindler Bernard Madoff told colleagues how to dance around questions from the Securities and Exchange Commission, according to an audiotape and transcript released Wednesday by the Massachusetts secretary of state's office.

"You know, you don't have to be too brilliant with these guys, because you don't have to be," Madoff said, referring to SEC investigators.

Authorities continue to unravel the far-reaching Ponzi scheme masterminded by Madoff that bankrupted some once-wealthy investors and forced many nonprofit groups and foundations to shut down or make drastic cuts.

At one point, Madoff interrupted the conversation to take another phone call. He returned and said, "I'm sorry. If I get any more solicitations for charity, I'm going to kill myself." Watch what else Madoff said »

Madoff, 71, was convicted of operating a Ponzi scheme and defrauding thousands of investors. He pleaded guilty and was sentenced to 150 years in prison in June. Prosecutors have said it was the largest investor fraud ever committed by a single person, totaling more than $50 billion in losses to investors.

Amit Vijayvergiya, chief risk officer of the Fairfield Greenwich Group, was included in the taped phone call. It wasn't immediately clear when the call was made.

Don't Miss

Fairfield Greenwich Group reached an $8 million settlement this week with the Massachusetts Securities Division in connection with the Madoff scandal. State regulators sued Fairfield for misleading investors and acting as a feeder fund to Madoff's Ponzi scheme.

With the settlement, Massachusetts dropped fraud charges against the company.

Thursday, representatives for the Fairfield Greenwich Group released a statement noting that the company asked the SEC for permission to speak to Madoff ahead of an SEC interview.

"Madoff's much-quoted line -- 'Obviously, first of all, this conversation never took place' -- was completely ignored by Fairfield Greenwich," the statement said. "Rather than take Madoff's suggestion to conceal the call, Fairfield Greenwich executives did the opposite. They told the SEC about the call and answered all questions accurately."

"Any implication that the Madoff call affected Fairfield's responses to the SEC is demonstrably false," the statement added. "There is no suggestion or allegation that anyone at Fairfield Greenwich tried to mislead the SEC."

In the phone call, Madoff cautioned his Fairfield associates not to know too much about how he was handling investments.

"You don't want to have that information because ... the commission, when they ask questions, they try and draw out information," Madoff said. "The less that you know how we execute ... the better you are."

Before running out of money in December, Madoff sent statements to victims, claiming that their investments had grown several times over. In actuality, he had stolen, not invested, their money. Investigators said they think he had run the scam since at least the 1980s.

In the phone conversation with Fairfield, Madoff said he called the shots at his company.

"I'm the only one that can make the decision. ... I'm the only one that pulls the trigger."

Madoff advised taking a casual approach in dealing with the SEC. He said, if asked, Fairfield executives should minimize the ties between the companies and not volunteer answers.

"They ask you a zillion different questions, and we look at them sometimes and we laugh, and we say, 'Are you guys writing a book?' "