LONDON—Xstrata PLC’s effort to seal a merger with Glencore International PLC has run into several unexpected obstacles since it was unveiled nine months ago. Now another hurdle is looming that could prove fatal.

Shareholders next week will vote on the merger using an untested ballot system devised to sidestep investor opposition to retention payments that Xstrata has proposed.

As next Tuesday’s vote approaches, people on both sides of the deal have expressed confidence that the merger will be approved. But behind the scenes, they are fretting that the complex voting procedure contains land mines that could blow up the deal even though most shareholders seem to be in favor of it.

The proposed merger would create a natural-resource company with a market value currently measured at $68 billion. It has traveled a rocky path since it was first proposed in February. Glencore originally proposed to pay 2.8 of its shares for each Xstrata share in a deal that would have awarded several senior Xstrata executives, including Chief Executive Mick Davis, millions of dollars for staying with the combined company for three years. That proposal called for Xstrata shareholders separately to approve the price and the retention payments.

When it became clear that both measures would be defeated, the two Anglo-Swiss companies set about restructuring the deal.

First, Glencore agreed to raise the price to 3.05 of its shares. In exchange Glencore demanded that its CEO, Ivan Glasenberg, get that post at the combined company instead of Mr. Davis. With Mr. Davis now expected to leave after six months, he has been eliminated from the retention pool, putting its size at around £140 million, or roughly $220 million.

Glencore’s new plan left Xstrata to address the other threat to the deal: the retention payments, which many U.K. shareholders said was a form of excessive executive compensation.

The solution Xstrata developed is a three-step voting proces that lets the deal go through without the retention payments. Xstrata shareholders will be asked to cast three votes: first, on a deal that includes the retention payments; second, on one that doesn’t; and third, on the retention payments themselves.

For the deal to pass, there has to be harmony between one of the first two votes and the third. If, for example, only the first resolution passes (approval with retention payments), then the retention payments also must pass for the deal to go through.

Adding to the difficulty is that the first two votes are subject to approval by 75% of the Xstrata shares that Glencore doesn’t already own. The threshold on the third resolution is 50%. Glencore already holds a 34% stake in Xstrata.