PORT-AU-PRINCE, Haiti (AP) — Back from exile, former strongman Jean-Claude "Baby Doc" Duvalier lives in a villa in the hills above Haiti's capital. His son serves as a consultant to the country's new president, Michel Martelly, while others with links to Duvalier's hated and feared regime work for the administration.

Duvalier himself is rumored to be ill and appears too frail to return to power. But for many Haitians who remember the ex-dictator's brutal rule, the rise of his loyalists to the new president's inner circle triggers suspicions about where Martelly's loyalties lie.

Such developments might be shrugged off in many countries, but not in Haiti, where much of the political establishment for the past 15 years has consisted of people associated with the mass uprising that forced "Baby Doc" to flee the country for France in 1986.

Now, a former minister and ambassador under the regime has been a close adviser to Martelly and on Thursday was nominated to the Cabinet. At least five high-ranking members of the administration, including the new prime minister, are the children of senior dictatorship officials.

Sen. Moise Jean-Charles said he and others who lived through those years are uneasy that Duvalierists are aligned with a president with no previous political experience and a history of supporting right-wing causes.

"They've been nostalgic for 25 years," Jean-Charles said of Duvalier's supporters. "And now, they're back in the country and back in power."

Martelly's powers will be at least partly held in check because his opponents control both houses of parliament.

Nonetheless, Jean-Charles, an ex-mayor under former President Jean-Bertrand Aristide, has taken his concerns to radio stations and the senate floor. Human rights advocates have echoed similar warnings, especially after a raucous protest staged by Duvalier supporters last month disrupted a news conference calling for the ex-dictator's prosecution.

"There's a lot of worry," said Haitian economist and sociologist Camille Chalmers. "The political circle is made up of Duvalierists."

Martelly spokesman Lucien Jura told The Associated Press that the appointments were based on individual qualifications rather than political affiliation.

"As President Martelly said before, he's not excluding," Jura said. "If the citizen is competent, honest and has good will ... regardless of the political sector he's in, he's welcome."

The new government includes a few veterans from Aristide's government, including Mario Dupuy, a communications adviser who was chief spokesman during Aristide's second term.

Martelly met with Aristide and Duvalier on Wednesday in an effort to reconcile differences between the former leaders and their followers. The day before he met with Prosper Avril, an army colonel who overthrew a transitional government in 1988 and resigned two years later amid protests.

"It's time for us again to be one nation, stand behind one project," Martelly told the AP outside the plush home where Duvalier is staying.

While running for office, Martelly pitched himself as a populist even if he later imposed taxes on remittances and phone calls from abroad to help pay for the free schooling of 772,000 children. He's also pledged to build housing and create jobs for some of the half million people still homeless nearly two years after an earthquake devastated the country.

While Martelly hasn't publicly voiced any support for Duvalier, he's addressed some of the top priorities of Duvalier's relatively small political base since taking office in May.

Last month, he proposed to restore the country's disbanded army in addition to award back pay to former soldiers dismissed by Aristide in 1995. Duvalier relied heavily on the military to crack down on internal dissent.

The proposed force will patrol Haiti's porous borders and provide relief during natural catastrophes as well as revive an intelligence unit that the CIA created after Duvalier's ouster to combat cocaine trafficking. That unit, the National Information Service, will assume a new role of fighting terrorism threats, mafia networks and "extremist" organizations.

Martelly told reporters Thursday that he is determined to restore the army even if he doesn't have the backing of others. It will be a modern force that won't hatch coups, he said.

"I don't want people to think that I won't start without them," Martelly said at the National Palace. "I will start without them if they don't start moving."

Critics say it would be better to improve the police force, which is more likely to remain independent.

"He can't control the police so he's trying to create his own force," Jean-Charles said.

Adding to the worries, Martelly hasn't pressed for the prosecution of Duvalier, who has been accused of looting the treasury and torturing and killing political opponents during his 15-year rule. Martelly has said it's up to the judiciary to handle Duvalier's case.

What's sparked the most concern has been the personnel picks of the musician-turned-president.

In his first months in office, Martelly turned to people such as Daniel Supplice, an ambassador and former minister of social affairs under Duvalier who was nominated Thursday to be minister for Haitians living abroad. Supplice, who led Martelly's transition team, hasn't been directly tied to the abuses associated with the Duvalier regime.

Martelly's also tapped the children of Duvalier officials including Prime Minister Garry Conille, a former chief of staff to Bill Clinton in the ex-president's role as U.N. special envoy to Haiti. Conille's father, Serge, was a minister of sports and youth for the dictatorship.

Conille declined to discuss his father and the family ties to Duvalier when asked by an AP reporter, responding, "I would expect a much more intelligent question from somebody like you."

Other picks with Duvalier links include Martelly's senior advisers Thierry and Gregory Mayard-Paul, brothers whose father Constantin Mayard-Paul was a lawyer for Claude Raymond, a feared army lieutenant general under "Baby Doc." Thierry Mayard-Paul was nominated Thursday to serve as minister of interior, local authorities and national defense in Conille's government.

The entire Cabinet still has to be approved by both houses of Parliament.

Raymond's son, Claude Jr., recently joined the administration as deputy director general for immigration. Josefa R. Gauthier, whose father Adrien was a diplomat under "Baby Doc's" regime, is the director general for the government's Fund for Economic and Social Assistance.

The most prominent tie is Francois Nicolas "Nico," Duvalier's 28-year-old son, who is a consultant to Martelly.

To be sure, Haiti's political ranks have been a revolving door since Jean-Claude Duvalier and his predecessor, father Francois, exiled thousands of professionals and shrank the talent pool from which governments draw qualified workers. Even Aristide, who helped lead the movement to oust the dictatorship, had a few Duvalierists in his administration.

But Martelly has hired more officials from the former regime than his last two predecessors.

The resurgent Duvalier movement made an assertive public appearance during last month's news conference organized by Amnesty International to discuss the stalled criminal investigation into "Baby Doc."

As a representative of the human rights group tried to speak, Duvalier supporters yelled into the microphones of journalists and shouted him down.

"You're trying to create a civil war in this country," Reynold Georges, a lawyer for Duvalier, told an Amnesty representative at the news conference. "If he needs to be tried, he will be tried."

(Copyright 2011 Associated Press. All Rights Reserved. This material may not be published, broadcast, rewritten, or redistributed.)