The farming minister David Heath has said protests against live animal exports at Ramsgate in Kent may be getting in the way of animal welfare.

He told the Guardian he was worried that government inspectors could not implement all checks because of the level of protests, which often draw dozens of campaigners to the port for sailings.

"We sometimes take actions that are not necessarily in the best interests of animals," he said. Heath also said the RSPCA, which strongly opposes the live export trade, "needs to make a choice over whether they are a fringe campaign group or a responsible organisation working with us in partnerships on animal welfare. They cannot do both."

Asked why Thanet district council should not be able to decide it did not want to deal in live exports, Heath said: "We have no responsibility to ensure there is a particular port. We have a responsibility to maintain the law, which says there must not be constraints on particular types of trade within the EU. We have no power to stop the trade."

He said: "We may prefer animals were slaughtered closer to the point of production, we may prefer ports with better facilities than Ramsgate were used, and we do. The fact is [the trade] is legal and there is nothing we can do to intervene on that."

Asked why the new measures only insisted that a government vet was within an hour of the port so that he or she could assist inspectors over welfare concerns or an emergency, Heath said animals were always checked by a vet before they were transported from farms. "We don't want to keep these animals hanging around at port side, which is a potentially stressful situation, any longer than we have to."

There was no mention of additional holding facilities nearer the port than animals' original farms in case sailings did not go ahead when the government announced the tougher measures. Heath said the government had "contingency farms nearby".

The challenge to the RSPCA, which intends to take legal action in a bid to halt the trade, is likely to further inflame the row between campaigners against exports and the government. Thanet council notifies the charity of shipments so the RSPCA can make its own welfare checks as well as government inspectors. Other protesters including Kent Action Against Live Exports have a network of members who watch lorry traffic.

The RSPCA, which the government says has no statutory role at the port, reacted angrily. Its chief executive Gavin Grant said: "Rather than criticising RSPCA Inspectors for doing their job, the Minister should focus on ending the suffering of thousands of animals being shipped in unacceptable conditions to an uncertain fate.

"Having been at the port myself, I'd be delighted to join the minister at Ramsgate again himself. He will see, as I did, that Government inspectors have ample opportunity to thoroughly examine every lorry and the state of the animals on board. They chose not to do so or to co-operate with RSPCA Inspectors to help ensure the welfare of the animals. This is a vile trade that harms animals and must end."