Many Boomers have visions of encore careers and even starting businesses when they "retire" from their current careers. And they are becoming entrepreneurs in record numbers.

There is a bold group of entrepreneurs who are financing their start-ups with their retirement savings. And there are companies that specialize in helping people do that.

But while people may be starting their dream businesses, many financial advisers say it's just a bad idea.

"As far as dipping into a retirement fund to bankroll a startup, I would generally not recommend that," says financial planner Mark VandeVelde, wealth partner with Hefty Wealth Partners in Auburn, Ind. "It is a tremendous amount of risk."

"You'd be surprised at the amount of people who see the IRA or 401(k) as a savings account," he says. "That is a dangerous approach."

But Kathleen White, who used $350,000 from her retirement account to launch a Two Men and a Truck franchise in Virginia Beach, Va., has no regrets.

"It's really interesting, I'm normally not a risk-taker," she says.

"This is such a strange thing," says White, 52. "I was not worried. I'm still not worried. I did a lot of homework. I've been meticulous. It took a lot before I decided to do this.I made up my mind that if I did it, I was all in."

Jania Bailey, president of FranNet in Louisville, Ky., a franchise broker, works with people who want to fund business opportunities. The company helps clients explore funding various sources.

One of those options is the self-directed 401(k) or IRA option. How it works: You have a business or franchise you would like to invest in. You buy stock in your new company with the IRA or 401(k) proceeds. Your new company is owned by and capitalized by your retirement fund.

"It's pretax investing," she says. "This is very different from withdrawing. It's not a loan. There are no monthly payments or penalties. Taxes are deferred. And you are beginning your new business virtually debt free."

"There are many firms that do this. We caution people to make sure you are using a highly qualified firm, someone who has been doing this for a while."

That's how Tara Bailey, 34, (no relation to Jania) financed her senior home care and staffing franchise, Right at Home, in Marietta, Ga. The former retirement consultant is not really worried about using her retirement to fund her business.

"My husband still maintains his full-time job," she says. "We had started early in retirement. I feel confident we have enough time to build that back up. Our main goal was to get the business up and running."

White is a lot closer to retirement. But still, she's not worried. She says she she has confidence in the Two Men and a Truck franchise because she had worked with the parent company previously, when she worked in economic development for the state of Michigan.

"I didn't feel a lot of concern about," she says. "I'm very confident this will be a successful venture."

She says her financial adviser was cautious, but ended up helping her find the company that would help do the 401(k) rollover.

Would she recommend that way of financing to others? Yes, she says, as long as they did their homework.

In fact, Jania Bailey says a survey of companies assisted by FranNet over the past five years showed that 85% were still operating. "Compared to people who start a business from scratch, that is very strong," she says.

Still, Jean Setzfand, vice president for financial security at AARP, says using your retirement to finance a business is a "risky proposition." She says she is "adamantly" against it for two main reasons.

"One is if you think about everything experts say about investing your retirement assets, you want to diversify risks," she says. "You want to place your assets in such a manner that you are not putting all your eggs in one basket. When you take your 401(k) assets and invest in a small business, that's putting the risk in hyper drive."

Second, she says, using your 401(k) keeps you from going through the process of getting a loan. That process, she says, forces you to think through your business plan, especially your revenue stream.

"We're definitely bullish on older individuals considering entrepreneurship as a viable option for earned income," Setzfand says. "But the way to pursue it is not to bet your retirement assets on that venture."

"There are people out there who have a huge appetite for risk, and no matter what their financial planner says, they will make the decision," says VandeVelde. "It is up to them to make the decision. But I want them to know the alternatives and the consequences."

Still, White says she has no regrets.

"I was an average Joe. I didn't have huge resources to pull from. This was an avenue for me to pursue a dream of owning my own business without going deeply in debt," says White, noting she doesn't have a bank loan.

"This business will be my retirement fund some day," she says. "By the time I am ready to retire, I'll be fine."