Antonio Jose Magalhaes de Nelo–whose farm near the town of Cafelândia, in São Paulo state, has been producing oranges for 25 years–has uprooted more than half of his 166,000 trees this year. He said the cost of growing and harvesting a 90-pound box of the fruit is about 3 Brazilian reais ($1.40) more than what he gets paid for the box.

“This sector is impossible now. Yesterday I did the numbers and figured out I paid BRL30,000 last year to grow and sell my oranges,” said Mr. de Nelo, 55, who is switching to sugar cane.

De Nelo is one of many Brazilian orange growers who are giving up on citrus and turning to other crops, usually sugar cane. Fewer trees could mean smaller global supplies of frozen concentrate orange juice, which is traded on the ICE Futures U.S. exchange.

Brazil is the world’s largest exporter of orange juice. It shipped 1.2 million metric tons of juice in 2012, according to the U.S. Department of Agriculture, almost 12 times more than the U.S., the second-largest exporter.

While the rising costs of labor and chemicals are a factor, orange growers blame the drop in domestic prices for pushing them out of the citrus business.

The average price offered by industrial buyers in Brazil has fallen nearly 60% in a little more than six years, from BRL15.46 for a box of oranges in January 2007 to BRL6.50 in May of this year, according to the University of São Paulo’s Center for Advanced Studies on Applied Economics.

The cost of producing a bushel of oranges and delivering it to the juice makers rose to BRL14.77 for the 2013-2014 growing season, federal crop agency Conab estimates, up from BRL13.90 in 2010, when the agency first started calculating the figure.

Part of the reason prices are low is consecutive bumper crops in Brazil in the previous two growing seasons boosted world supplies and pushed prices in the world market down 31% in 2012. But growers say Citrosuco SA Agroindústria, Sucocitrico Cutrale Ltda. and Louis Dreyfus Commodities Brasil SA–the three Brazilian companies that sell 98% of the juice exports—are also setting low prices for the oranges they buy.

Together, the three companies account for about 40% of Brazilian orange production.

Citrosuco and Cutrale didn’t respond to requests for comment. Louis Dreyfus Commodities referred questions to Ibiapaba Netto, a spokesman for CitrusBR, a trade association representing the three juice exporters, who said what the growers do with their land is their own decision.

São Paulo state’s Agricultural Economy Institute, or IEA, says the number of fruit-bearing orange trees in the state, the biggest producer in Brazil, fell 8.3% to 185.5 million in 2012. That’s the second-lowest number in the past five years. That decline, though, includes trees that have been cut down due to age or disease.

”If (reports of farmers uprooting trees) are true, and if the information starts to filter out into the trading community, that’s definitely bullish” for orange-juice futures prices, said Joe Nikruto, senior commodities broker at RJ O’Brien Futures in Chicago. Frozen concentrate orange juice fell almost 2% to $1.4570 a pound in late trade Thursday.

However, declining demand for orange juice may rein in any rally in futures prices. Orange juice consumption around the world fell by 6.7% from 2003 through 2010, according to CitrusBR. In the U.S., a major consumer of orange juice, total gallons sold have fallen 14% from December 2010 to May 2013, according to Nielsen data published by the Florida Department of Citrus.