The death toll from the collapse of the Rana Plaza factory near Dhaka has risen to over 400 people. Photograph: Reuters

The EU is considering trade action against Bangladesh, which has preferential access to EU markets for its garments, to pressure the country to improve safety standards after a building collapse killed more than 400 factory workers.

The death toll from the collapse last week of the illegally built Rana Plaza near Dhaka rose to 411 on Wednesday, and about 40 unidentified victims were buried. There were about 3,000 people inside the complex, which was built on a swamp, when it collapsed. About 2,500 people were rescued, many injured.

Duty-free access offered by western countries and low wages have helped turn Bangladesh's garment exports into a €14.4bn (£12.2bn) a year industry, with 60% of clothing going to Europe.

"The EU is considering appropriate action, including through the generalised system of preferences, through which Bangladesh receives duty-free and quota-free access to the EU market," Ashton and De Gucht said. "The sheer scale of this disaster and the alleged criminality around the building's construction is finally becoming clear to the world."

Any action by the EU on Bangladesh's duty-free and quota-free access would require the agreement of all member states and could take more than a year to implement.

'Turning up the diplomatic heat'

With local anger growing at the country's worst industrial accident, a delegation from the International Labour Organisation (ILO) met the Bangladeshi prime minister, Sheikh Hasina, in Dhaka to offer support and press for action to prevent such incidents.

The EU had already urged Bangladesh to adhere to ILO standards in January after two earlier factory fires, including one last November in which 112 people died. An EU official said the latest statement was "a shot across the bows". "We want to turn up the diplomatic heat on them and get them to sit down and discuss this with us."

About 3.6 million people work in Bangladesh's garment industry, making it the world's second-largest apparel exporter behind China. The industry employs mostly women, some of whom earn as little as €29 a month.

The Bangladeshi prime minister warned factory owners they would have to take care of their workers. "You will have to ensure workers' fair wages, allowances and other rights … you must look after their workplace safety if you want to do business," she told a discussion forum.

The April factory collapse in Bangladesh was the third deadly incident in six months to raise questions about worker safety and labour conditions in the poor south Asian country, which relies on garments for 80% of its exports.

In the year to June 2012, Bangladesh's garment exports to the EU rose to €8.6bn from €8bn a year earlier, according to Bangladesh's commerce ministry. Germany is the main EU market, followed by the UK, Spain and France. Bangladesh's next biggest garment export market is the US.