Ever get the impression that your financial adviser just isn't that into you? It could be time to move on.

Sheryl Garrett, founder of the Garrett Planning Network, a network of fee-only financial advisers and planners, says many people stick with their advisers for years longer than they should. "Even when you put the time and energy into finding the right person, it doesn't always last," she says, comparing the client/adviser relationship to a marriage.

However, clients have a lot more control than many people realize. Garrett suggests that in a lot of cases, "if the adviser is not working out for you or is not meeting your expectations, you should express what you want and give them an opportunity to correct it."

Reasons for leaving a financial adviser include:

Lack of communication

As with marriage, the issue often boils down to a breakdown in communication. In fact, a Spectrem Group report released in 2011 found that the No. 1 reason millionaire investors switch advisers is that they're not getting a prompt response to phone calls. Lack of response to your inquiries is one thing, but Garrett says you shouldn't expect your adviser to check in with you weekly or monthly, especially if you're paying him or her on an hourly basis. Still, you should be getting statements regularly even if you aren't getting personal calls.

Mismatch of personality or values

Adam Koos, a certified financial planner and president of Libertas Wealth Management in Dublin, Ohio, points out that sometimes clients and advisers don't work well together because of differing personalities. Of course, this can go both ways. "Just like the client has the right to pick and choose who they work with from a values standpoint, I weigh my preference with whom I want to work with," he says. Political and other values can impact that relationship. For instance, if your adviser's strong opinions on taxes or the future of Social Security make you uncomfortable, that could sour the relationship.

Frustration over fees

Many investors don't understand advisers' fee structures, according to Garrett, so as the relationship evolves, they may not feel they're getting their money's worth. "It could be a very legitimate situation that they've benefited greatly from the work with the adviser in the early months or years, " she says. "But now as things are clicking along as expected, the adviser may still have a fairly substantial fee but the client is just receiving maintenance." Garrett says it's fair to question fees that you don't understand or ask if your adviser is willing to revisit fees.

Poor performance

Even the best advisers can have a less-than-stellar quarter, but if your assets are consistently underperforming over a longer period, it could be time to move on. According to Koos, the average "up" market lasts about three to four years, while the average downmarket lasts about one year. "Two to three years is more than enough time to see how a financial adviser has done through both down and up markets," he says. Garrett suggests getting a second opinion from a fee-only adviser on an hourly basis if you aren't sure how your investment fees and performance measure up or if you sense you're being sold products that aren't in your best interest.

Ethical or legal issues

The last thing you want is to entrust your investments to the next Bernie Madoff, so if something seems amiss, speak up. "Whenever you feel like you're being pitched an investment that seems a little 'off book,' if the paperwork seems strange, or if they try to make it sound as if the investment is so exclusive that it needs to be kept on the hush-hush, I'd run as fast as you can," Koos says.

A major life event

According to Fidelity Investments, 70 percent of widows leave the family adviser within a year of a husband's death. People also switch advisers when they move, marry, divorce, or switch jobs -- "whenever there's emotion around money," as Koos puts it. Sometimes these life changes also cause clients to seek different expertise. "Certain advisers specialize in wealth accumulation, where most of what they do is help people grow their wealth, so once you retire, you may want someone different," adds Koos.

If and when you decide to move on, a letter is often the best way to communicate that decision, because many people are uncomfortable dumping their adviser in person or over the phone. As Koos points out, if you call the adviser, he or she will be likely try to convince you to stay. A letter is also a good way to document a complaint if you have one.

Those letters are courteous but rare, according Garrett. More often, advisers find out they've lost a client when they get a request to transfer funds. "They have been known to call the client and say, 'Mary, what did I do wrong?'" explains Garrett. "I let clients use me as the bad guy, saying 'we have chosen to work with Sheryl going forward. If you have any questions, you can call her.'"

To ease the transition, make sure your new financial adviser has all the statements from your old firm. Koos recommends filling out the new paperwork in person so you don't miss a form or signature line.