Malaysia’s state-owned energy company said Tuesday it was ending its multibillion-dollar plan to build a natural-gas export terminal on Canada’s west coast, saying prolonged low natural-gas prices doomed the project.

The decision marks a major setback in Canada’s attempt to build up its capacity in the liquefied natural gas market, which is now dominated by the U.S., Australia and Qatar. It also brings an end to one of the biggest proposed energy ventures in recent years: The project had a total price-tag of 36 billion...