The New York Stock Exchange agreed to settle allegations it broke rules designed to protect investors, the second such pact in less than two years as the top U.S. securities regulator tries to ratchet up the policing of stock markets.

The Big Board, along with affiliated exchanges and a broker, agreed to pay $4.5 million to settle civil charges by the Securities and Exchange Commission that they "repeatedly" broke their own...