TORONTO — Barely 12 days before the deadline for a final agreement, Canada suspended free-trade talks with the United States on Wednesday and ordered its chief negotiator home for consultations when American officials refused to concede on the key Canadian demand.

Prime Minister Brian Mulroney told the House of Commons that he authorized suspension of the 16-month-long negotiations after Simon Reisman, head of the Canadian team, reported that the Americans "were not responding on elements fundamental to Canada's position."

Canadian government officials said this meant that the United States has refused to meet Reisman's demands for an impartial dispute-settling mechanism that would be binding on both nations.

Not in 'National Interest'

Ambassador Reisman was concerned about some fundamental issues," Mulroney said. "As I told him at the outset: 'If at any time you felt the national interest requires you to break off and come home, you can do that because we will never sign an agreement not in Canada's national interest.' "

American negotiators said that they had not suspended the talks and were willing to keep going.

"We did not break off the talks," said U.S. Trade Representative Clayton K. Yeutter, "and regret that the Canadian government has chosen to do so."

Chief U.S. negotiator Peter Murphy, under pressure from Congress, has consistently declined to accept any proposal that would take away from Congress its power to set trade policy. That includes the authority to decide whether any country--Canada included--is using unfair tactics in selling goods in the United States.

Binding Arbitration

Before talks broke off, negotiators had discussed subjecting only a narrow list of goods to binding arbitration in the event of trade disputes between the two nations. For a much larger number of goods, each nation would be free to impose sanctions against the other in the event of unfair trade practices.

The American negotiators, sources said, believed that even this method encroached too much on congressional authority.

Canada is already the United States' largest trading partner, and the two-way trade between the neighbors is the largest in the world, approaching $150 billion this year. In 1986, Canada's exports to the United States totaled $68.9 billion and its imports were $55.6 billion.

Since Canada ships about 80% of its exports to the United States, accounting for 20% of its gross domestic product, the growing protectionism in the United States has been of profound concern to Mulroney.

Mulroney told a raucous session of Commons that the U.S. position was "deemed to be unsatisfactory." Nevertheless, the prime minister said, the talks could resume "when it is clear from the American side that matters of vital concern to Canada will be addressed in a satisfactory way."

'Meet Around the Clock'

In Washington, Yeutter insisted that "there are no differences that cannot be bridged" and said that U.S. negotiators "are prepared to resume talks and are willing to meet around the clock if necessary to complete an agreement by the Oct. 4 deadline."

Congress has agreed to give quick consideration to any agreement reached by Oct. 4. An accord reached after that date probably would fall victim to the delays that have bottled up other free-trade legislation in recent years.

U.S. officials left the impression that they consider Reisman's walkout to be a negotiating ploy and a move by Mulroney to bolster his image at home as a tough leader unafraid of the Americans.

"We had told them we were willing to work creatively to come up with something," said a Reagan Administration official who asked not to be identified. "But the fact is, they walked out 12 days early."

In Canada, hopes persisted that the talks remain alive. Reports circulated in Ottawa that Mulroney intends to contact President Reagan, probably by telephone, in an effort to change the U.S. position.

More on the Impasse

Furthermore, U.S. Treasury Secretary James A. Baker III and Canadian Finance Minister Michael Wilson are expected to discuss the apparent impasse in meetings Saturday in Washington, where the world's finance ministers are gathering for the annual fall meeting of the World Bank and International Monetary Fund.

But in Washington, Canadian sources close to the negotiations were pessimistic to the point of bitterness.

"We were hoping to see some real movement on trading rules, the principles to enforce the agreement," explained one official, who spoke on condition that he not be identified. "We wanted firm North American trading rules that would apply equally to both sides, and we wanted a tribunal that would apply those rules objectively.

"About 10 days ago we thought we had some movement on the U.S. side, so this week it came as a major surprise to us that the Americans seemed to have toughened their stand and become more rigid.

"We've not given up all hope, but it's up to them. It's gloomy, all right."

Rich U.S. Market

Canada, citing a 1981 speech by President Reagan calling for a North American trade compact, had asked for the trade negotiations in 1985. The issue became so important to Mulroney that he has virtually staked his political future on a successful pact that would guarantee Canadian access to the huge and rich American market.

He told Commons that a successful trade arrangement would "create thousands of new jobs for Canadians. . . . There is no doubt . . . that we will continue to seek a comprehensive trade agreement."

Kenneth Freed reported from Toronto and Oswald Johnston from Washington.