HONG KONG (AP) -- Hong Kong police arrested more than 500 people who refused to leave a street in the city's financial district Wednesday, a day after tens of thousands of people joined a massive march to push for democracy, free from China's interference.

The march is an annual affair to mark the anniversary of the day China took over Hong Kong from Britain on June 1, 1997 with the promise to give the city a high degree of autonomy for 50 years. But there is growing unease among its residents -- especially the youth -- that the Western-style civil liberties they've know all their lives are being eroded as the freewheeling capitalist enclave comes under Beijing's hard-line communist rule.

The fears are only going to be heightened following the pre-dawn crackdown by the Hong Kong police, who normally do not have an antagonistic relationship with the people, unlike the security forces in mainland China.

Police said 511 people were arrested for unlawful assembly in the Central business district and preventing police from carrying out their duties. After warnings failed to dislodge the protesters, the police moved in and forcibly removed them. Some went willingly while others lay down on the street with arms locked, and had to be physically carried away one by one. The protesters had vowed to stay until 8 a.m., just before the height of rush hour began, but the police started moving in to evict them at about 3 a.m.

Those arrested were mostly students who had decided to occupy Chater Road after taking part in Tuesday's rally, which police said attracted 98,000 people at its peak. Organizers said 510,000 people turned out, the highest estimates in a decade. Hong Kong researchers put the number at between 154,000 and 172,000.

Whatever the numbers, the march and the vehemence of opposition is certain to raise the alarm in Beijing, which is vowing to restrict the democracy it has promised Hong Kong.

China's Communist leaders have pledged to allow Hong Kong residents to vote for the leader by 2017. However, they've rejected calls to allow the public to name candidates, insisting instead that they be vetted by a Beijing-friendly committee like the one that has hand-picked all leaders since the handover.

Also, three weeks ago, the Chinese government released a so-called white paper that said Hong Kong's high degree of autonomy is not inherent but is authorized by the central government in Beijing.

Anger against those moves was on display during the march Tuesday when peaceful crowds carried banners and posters urging democracy. They steady throng of people walked in sweltering heat and occasional rain from Victoria Park, through a broad boulevard lined with skyscrapers to the financial district. Thousands of police kept watch and ordered the city's iconic trolleys to shut down along the boulevard to reduce overcrowding.

"After seeing the white paper's content, we should be worried," said Jeff Kwok, 28, an export firm employee at Victoria Park, where six soccer fields and surrounding areas were jammed with people. Beijing is "trying to tell the Hong Kong people that ... Hong Kong is just one of their regions. They're trying to tell us they have absolute power to rule us."

Kwok complained that Beijing doesn't respect the principle of "one country, two systems" under which Hong Kong is allowed to retain control over much of its own affairs.

Another protester, Kennie Chan, lamented that Beijing was less restrained now in exerting its influence over Hong Kong.

"In the past, it seemed like they were doing it step by step, but now, it's obvious that they cannot stand Hong Kong people. We are not obedient anymore, and are resisting more and more," said the 30-year-old, who works as a stage manager.

Ahead of the rally, one group of protesters burned a copy of the white paper outside a ceremony attended by officials to mark the handover.

The protest comes days after nearly 800,000 residents voted in an informal referendum aimed at bolstering support for full democracy. Beijing denounced the referendum as a political farce.

Leung Chun-ying, Hong Kong's Beijing-backed leader, tried to soothe tensions, saying in a speech that he'll do his "utmost to forge a consensus" on implementing universal suffrage on schedule. But the government later released a statement saying it is unlikely that public nominations will be allowed because it's legally "highly controversial."