NEWCASTLE, Australia—Rio Tinto PLC failed to overturn a court decision blocking the expansion of one of its largest Australian coal pits, in the latest blow to company mining operations already bruised by weak demand and commodity fuel prices.

The 600 million Australian dollar (US$558 million) expansion of Rio Tinto's Warkworth open-cut mine has been in limbo since the New South Wales Land and Environment Court last year ruled it...