Manila -- Police in Malaysia were in a standoff late Thursday with at least 80 militiamen from the Philippines who were seeking to stay by right of historic claims on the island of Borneo, police said.

The men, who arrived by boat Tuesday in Sabah state, on the Malaysian side of Borneo, said they were descendants of the leaders of the Sultanate of Sulu, an area ruled from the southern Philippines that in the 18th century included swaths of the island.

Lahad Datu, the small village in eastern Malaysia where the men are, is less than two hours by speedboat from the southern Philippines, where several violent rebel groups operate. But Tan Sri Ismail Omar, the head of the Malaysian Royal Police, told reporters Thursday that the men denied links to any militant organization in the Philippines.

The police chief said that the situation remained peaceful, and that negotiations to get the men to return to the Philippines were continuing.

Malaysia's prime minister, Najib Razak, who was visiting Sabah on Thursday, told reporters that the men were surrounded, according to the Malaysian state news agency, Bernama.

"In terms of strength, we have the upper hand in combat power to arrest them, but the government opts for negotiation to break the stalemate so that they leave peacefully to southern Philippines," Bernama quoted the prime minister as saying.

Ricky Carandang, a presidential spokesman in Manila, said the Philippines was trying to assess the situation and was in contact with security forces in Malaysia.

The area where the standoff is taking place is on the Malaysian-controlled part of the island of Borneo, and is less than 90 miles from the southern Philippine province of Tawi-Tawi, where several kidnappings have taken place in recent years, including the abduction of two European bird-watchers in January 2012.

In the nearby Philippine provinces of Sulu and Basilan, the violent rebel group Abu Sayyaf has kidnapped and sometimes beheaded Filipinos and foreign visitors.

The Philippines has a long-standing claim over parts of the island of Borneo, and there have been tensions in years past between the two governments.