The London Stock Exchange has become a takeover target after its proposed merger with Canadian rival TMX was ditched. The Toronto-based group indicated it was unlikely to win the necessary two-thirds support from its shareholders.

The collapse of the deal is a major setback for LSE chief Xavier Rolet. At a time of rapid consolidation of world exchanges – for example, Deutsche Börse has swooped on NYSE Euronext – analysts believe London is now vulnerable to a bid from a player such as Nasdaq, the US technology bourse that tried to acquire the LSE in 2007. Other possible suitors include the Hong Kong and Singapore exchanges.

The LSE's bid for TMX was trumped late on Wednesday by rival Canadian consortium Maple, playing the nationalist card, and was supported by large Canadian banks such as Toronto-Dominion. At the end of last year, British mining group BHP was forced to scrap a takeover of another Canadian company, Potash Corporation, after the government in Ottawa ruled the deal provided no "net benefit" to Canada.

Although Maple's offer for TMX was worth more than the LSE's and offered a "Canadian solution", its merger plan could be blocked on competition grounds.

TMX said a majority of shareholder proxy votes supported the LSE merger resolution. "However, it is clear that the two-thirds threshold required to approve the merger would not have been achieved."

Rolet said: "We are clearly disappointed that, despite a majority of both LSE and TMX Group shareholders voting for our recommended merger, the two-thirds approval threshold for TMX Group shareholders was not met and hence the merger will now not proceed."

He added: "Our group is in good shape and financially robust. Whilst the merger with TMX Group was an exciting opportunity for LSE, we continue to see other significant growth opportunities."

In terminating the merger agreement, TMX has agreed to pay a $10m (£6.3m) break fee to the LSE, and a further $29m if, within 12 months, it merges with Maple.

Simon Maughan, an analyst at broker MF Global, said: "The failure of the TMX deal will leave the LSE a target as investors have bought into the stock recently in the belief the TMX bid will fail and the LSE will end up on the block."

Arnaud Giblat at UBS said there was an obvious candidate waiting for the London exchange should the LSE's offer for TMX fail: "Nasdaq seems to want to bulk up and the LSE fits the bill."

Canada's main opposition party argued the LSE/TMX deal was not in the public interest and demanded the government hold a public inquiry. The left-leaning New Democratic Party said the arrangement, as structured, "was flawed and would lead to a foreign takeover of Canada's capital markets". It urged the federal government, which has a majority of seats in the legislature, to refrain from giving the deal its "rubber-stamp" approval.

Under Canadian law, any foreign acquisition or investment over C$312m (£200m) requires federal government approval and must pass a test ensuring it provides a net benefit to the country.

TMX operates cash and derivative markets for equities, fixed income and energy and owns both the Toronto and Montreal exchanges.