Hedge-fund and private-equity managers over the past year began piling into debt issued by troubled energy companies, hoping to profit off a reversal of oil’s slide. They raised billions of dollars for the effort, in many cases telling backers it was a once-in-a-generation chance to pounce. But crude has continued to fall, slamming the companies and many large investors who thought they had bought in near the bottom.

An energy fund at Magnetar Capital LLC is down 12% this year through the end of September after the more-than $ 14 billion hedge-fund firm invested billions of dollars in distressed oil and gas companies, according to investor documents reviewed by The Wall Street Journal. Brigade Capital Management LP’s main hedge fund is having its worst stretch since 2008 in part because of the $ 16 billion firm’s exposure to junk-rated energy companies, investor documents show. The same factors have helped put King Street Capital Management LP, a $ 21 billion firm, on a course for the first annual loss in its 20-year history, according to people familiar with the matter.

“A lot of hot money chased into what we believe are insolvent companies at best,” said Paul Twitchell, partner at hedge-fund firm Whitebox Advisors LLC, which he said steered clear of the trade. “Bonds getting really cheap doesn’t mean they are a good buy.”

Hedge-fund and private-equity managers trumpet their ability to spot beaten-down assets and to swoop in when others are wary. They have had mixed results lately, and the painful energy trade is among the bigger reasons, along with wrong moves ahead of the August devaluation of China’s yuan and a plunge in pharmaceutical companies due to concerns about drug-pricing practices.

That is because during oil’s deep decline in 2014, many firms including private-equity firm Carlyle GroupCG-3.54% LP had their eyes on an energy trade they hoped would be a layup, especially in an era of low interest rates: They could lend to and invest in cash-strapped energy companies, and book a profit when the market returned to its historical norms.

In March, Carlyle founder David Rubenstein said at a conference that he was looking to “buy now” on the theory that prices would rebound.

Such wagers haven’t panned out, at least not so far. Crude oil traded on the New York Mercantile Exchange has dropped 45% in the past year and is off 14% in 2015, as questions about China’s economic growth have weighed on commodity prices.

Meanwhile, the market for junk-rated energy debt has dried up. In the third quarter, the volume of such bonds in the U.S. issued by energy companies fell to the lowest level since 2011, according to data provider Dealogic.

Private-equity firms may be able to wait out the turmoil, as they aren’t due to pay back investors for years to come. Hedge-fund managers don’t have the same luxury. For years, hedge-fund managers have bemoaned the lack of trading opportunities in a period of relative placidity in the markets. Now that volatility is back, however, few firms have been able to cash in.

Hedge funds that primarily invest in distressed assets are down 4% on average this year, while hedge funds that focus on fixed-income investments are flat overall, according to researcher HFR. Both have lost money in each of the past four months. The Barclays Capital Government/Credit Bond Index is up 1% this year.

It has been a particularly harsh stretch for a handful of managers who specialize in complicated debt trades and tout their ability to be ballasts for investors in a variety of macroeconomic environments.

The biggest fund at junk-bond-focused Phoenix Investment Adviser LLC declined 24% through the end of August, according to investor documents. The $ 1.2 billion firm has posted losses in 11 of the past 12 months, the documents show. The firm told investors some of its biggest losses have come from the bonds of oil and gas producer Goodrich Petroleum Corp.GDP1.14%

Donald Morgan, founder of Brigade Capital Management. The firm’s main hedge fund is having its worst stretch since 2008 in part because of the firm’s exposure to junk-rated energy companies. Photo: Jin Lee/Bloomberg News

Even King Street, one of the steadiest performers in the hedge-fund world, has lost money for five consecutive months, according to people familiar with the matter. King Street has been weighed down by one of its biggest holdings, bankrupt Texas utility Energy Future Holdings Corp., the people said.

Despite the losses, many investors aren’t changing course. Mr. Peskind said he views the sustained pullback as an “unbelievable potential buying opportunity,” given the overall strength of the U.S. economy.

Magnetar is also looking to make new investments tied to energy’s fall. It has made back in October some of the 12% loss its MTP Energy Fund suffered during the first three quarters, a person familiar with the matter said.

Brigade fell more than 7% this summer, erasing its gains for the year and pushing it into negative territory in 2015, as positions in companies like Texas Competitive Electric Holdings Co. LLC came under pressure. In an investor letter, the firm lamented that companies were falling “despite no credit-specific news” and said its traders were buying more of some hard-hit energy companies.

Still, its peers continue to raise new money. Carlyle is still pitching investors on filling out a new $ 2.5 billion fund, according to an investor presentation. Its target: energy projects and companies.