Goldman Sachs Group Inc. is upending the way it does business with hedge funds, jettisoning less-profitable clients and increasing some fees on others as it adapts to new banking rules, people familiar with the matter said.

The bank has told hedge-fund clients that the regulations have forced it to set aside more capital, crimping profits at its prime-brokerage business, which executes and finances the funds' trades.