The abduction of Libyan Prime Minister Ali Zeidan in Tripoli on Thursday by heavily armed gunmen on the government payroll underscores the power militias wield in the North African nation two years after the ouster of dictator Moammar Gadhafi.

Mr. Zeidan was taken at gunpoint in a pre-dawn raid on the luxury Corinthia Hotel that he and other government officials use as a residence. He was released several hours later.

Secretary of State John F. Kerry condemned the abduction, saying Libyans “did not risk their lives in their 2011 revolution to tolerate a return to thuggery.”

French President Francois Hollande said Mr. Zeidan’s abduction reinforces France’s concerns about Libya.

It was not clear why Mr. Zeidan was abducted. But sources, who spoke on the condition of anonymity, said the reasons range from dissatisfaction over government corruption to anger over a weekend U.S. commando raid in Tripoli that resulted in the capture of al Qaeda suspect Nazih Abdul-Hamed al-Ruqai, known by his alias Abu Anas al-Libi.

The Libyan government has complained to the United States that the operation to capture al-Libi violated its sovereignty. Al-Libi is wanted by the United States in the 1998 bombings of the U.S. embassies in Tanzania and Kenya. He had a $5 million bounty on his head.

A militia known as the Operations Room of Libya’s Revolutionaries claimed responsibility for kidnapping Mr. Zeidan. The militia is paid by the government to protect Libya’s parliament.

The Libyan government relies on militias formed during the revolution against Gadhafi to provide security.

Some militias have refused to submit to government control, creating lawlessness in parts of the country. But the role of a militia on the government payroll in the abduction of the prime minister exposed the serious nature of the security challenges facing the government.

“The abduction of Mr. Zeidan has sent a clear message to everybody that it is not the government, but the militias that are in charge,” said Hassan El Amin, the former chairman of Libya’s parliamentary human rights committee. “This is also a clear message to the outside world that Libya is coming close to being a failed state.”

Mr. El Amin said the General National Congress had been “hijacked” by militiamen and their supporters who have been elected to parliament.

“As long as these militias are there, we cannot move forward,” said Mr. El Amin, who fled Libya in March in the face of threats to his life after he criticized militias for their unchecked power and flagrant violations of the law.

Human rights groups, including Amnesty International, have documented some of these violations, including illegal detentions and torture of detainees by militias.

“Libya’s transition continues to be undermined by militias, which act above the law with impunity,” said Frank Jannuzi, deputy executive director of Amnesty International USA. “Libya’s Ministry of Justice now says it controls some 37 prisons, but in many of these cases, militias are still running the show.”

The Libyan government’s efforts to bring the militias under its control have had mixed results.

“This has often occurred without proper training or vetting to screen out human rights violators, continuing the risk of human rights violations in the days ahead,” said Mr. Jannuzi.

Even as the government has been largely powerless against these groups, Libyans have often risen up against the militias.

The most prominent backlash followed the Sept. 11, 2012, terrorist attack on the U.S. diplomatic compound in the eastern city of Benghazi. Terrorists killed U.S. Ambassador J. Christopher Stevens, State Department officer Sean Smith, and former Navy SEALs Glen Doherty and Tyrone Woods.

Libyans stormed the base of Ansar al-Shariah, the group suspected in the attack, forcing it to abandon the city. But Ansar al-Shariah has since returned to Benghazi.