Around 100 human rights activists protested outside Japan’s de facto embassy in Taipei on Monday demanding that a member of a Japanese right-wing group be barred from leaving Taiwan after he was caught on security camera kicking a “comfort woman” memorial.

The footage showed Mitsuhiko Fujii, who is in Taiwan representing 16 right-wing groups from Japan, kicking the bronze statue outside the opposition Kuomintang’s office in the southwest city of Tainan after he visited on Thursday, asking for it to be removed.

Taiwanese, mainland Chinese, Korean, Filipino and other Asian women forced to work in Japan’s wartime military brothels were euphemistically referred to in Japan as “comfort women”.

The statue is the first such memorial in Taiwan and was installed on August 14, the International Memorial Day for Comfort Women, by a local association and the KMT. Former president Ma Ying-jeou, who has long demanded Japan apologise over the comfort women issue, took part in the ceremony to unveil the statue.

The KMT office said it was not aware of the incident, which it described as “humiliating”, until media reports emerged. Office director Hsieh Lung-chieh, who is also a Tainan councillor, said he then reviewed the surveillance footage and confirmed Fujii had kicked the statue as he was leaving the premises.

“It is inexcusable and unbearable for [Fujii] to come here and trample on the dignity of Taiwanese people, insulting our grandmothers who were forced to serve as comfort women [by Japan],” Hsieh said.

On Monday, protesters – some of them wearing white masks – gathered in front of the de facto embassy, the Japan-Taiwan Exchange Association, waving banners calling for an apology, and saying the incident had seriously hurt the feelings of Taiwanese.

The activists demanded that Taiwanese authorities stop the man from leaving the island until he apologised for his act – and asking for him to kneel in front of the statue when he did so.

They also called for the Japanese representative to Taiwan, Mikio Numato, to apologise to the public for the incident, and for Taiwan’s envoy to Japan, Frank Hsieh Chang-ting, to lodge a formal protest over the act, which they described as “insulting”.

The protesters briefly scuffled with around 50 police officers trying to stop them from forcing their way into the building, and after a near two-hour stand-off a senior Japanese official accepted their protest letter. Some of the activists were from the Tainan City Association for Comfort Women’s Rights, which arranged for the statue to be installed.

The statue ignited heated debate between the KMT and the Japanese-friendly ruling Democratic Progressive Party, with the DPP accusing the opposition of trying to stir up resentment against Japan in a bid to win voters ahead of November’s local government elections, which are seen as a prelude to the 2020 presidential poll.

At the unveiling ceremony, former KMT leader Ma criticised President Tsai Ing-wen’s government for failing to push for Japan to issue a formal apology to Taiwan for forcing women into sexual slavery during the second world war.

The monument also drew a rebuke from Japan, with Chief Cabinet Secretary Yoshihide Suga describing the installation as “extremely disappointing”. In Taipei, the de facto embassy expressed Tokyo’s regret over the statue going up, saying it and others like it around the world went against the Japanese government’s position on and efforts to resolve the issue and calling for it to be removed.

The latest incident is expected to further inflame the row between Japan and Taiwan over wartime sexual slavery and will add to pressure on Tsai’s government, which critics say has bowed to Tokyo and failed to do anything to protect the interests of Taiwanese.

So far, the government has insisted that it has nothing to do with the statue and it was put up by a local group and the KMT. The foreign ministry said Taipei was closely watching the issue and it remained government policy to fight for the rights and dignity of the comfort women.

Since it was installed a month ago in a public space near several buildings from the Japanese colonial period, the statue has been visited by more than 500 people every day, including from Japan, according to Hsieh’s office.

The numbers on how many Taiwanese women were forced into sexual slavery for the Japanese military differ – the KMT puts it at 200,000 to 300,000, while the Taipei Women’s Rescue Foundation says it is around 2,000, based on historical records. There are only two identified survivors still alive in Taiwan.

Japan set up the Asian Women’s Fund in 1994, offering compensation and a letter of apology from the prime minister to the victims, but many former comfort women – including those from Taiwan – refused to accept the money on the grounds that the Japanese government has yet to address its legal responsibility on the issue – something Ma also called for.