Nasdaq OMX Group Inc. may be forced by securities regulators to upgrade its trading systems in the wake of last month’s glitch-ridden stock sale by Facebook Inc.

The Securities and Exchange Commission is investigating what caused the mishaps that plagued the Nasdaq Stock Market during Facebook shares’ debut on May 18 and the effects on brokers and investors who lost money.

As part of the deepening inquiry, regulators are weighing demanding that Nasdaq agree to revamp its processes for developing, changing, testing and implementing the computer code used in initial public offerings and other exchange functions, according to people familiar with the investigation. The SEC hasn’t decided yet whether to take any enforcement action against the company.

A regulatory request to improve its systems would be a rebuke for Nasdaq, a company that has long cultivated an image as a leader in exchange technology. It also could add to the exchange’s costs.

The SEC review comes as the exchange group prepares to submit to the SEC its plan, expected next week, for compensating brokers who claim losses on trades it failed to execute properly, and helps keep the spotlight trained on Nasdaq’s performance during one of the most hotly anticipated IPOs in years and on its CEO, Robert Greifeld.

Nasdaq’s emergence as an all-electronic trading forum helped transform the exchange industry, and its reputation for cutting-edge systems has attracted the stock listings of generations of Silicon Valley’s best-known technology companies. Facebook was supposed to burnish that reputation.

Nasdaq said this month that it had hired International Business Machines Corp. to review its technology systems. The exchange group’s executives are also reviewing its management structure, focusing on the operations and technology areas overseen by Anna Ewing, said people familiar with discussions inside Nasdaq. Some executives said her responsibilities, which include overseeing Nasdaq’s own trading technology as well as products it sells to brokers, would be too broad for any manager.