Bats Global Markets agreed to pay $14 million – the highest fine levied by a regulator against a stock exchange – to settle allegations that two exchanges it bought last year failed to disclose to investors important information about how their markets functioned.

The settlement Monday brought to a close a more than three-year investigation by the Securities and Exchange Commission into how Direct Edge, which merged with Bats last year, gave some high-speed traders an advantage over others by not providing details about certain order types, according to the SEC.

Bats didn’t admit or deny the allegations as part of the settlement.

The probe fueled a debate among regulators, investors and the financial industry about whether the markets have become too complex and geared toward sophisticated traders who use speed and other advantages to get an edge.

Order types, a set of instructions an investor uses to trade on exchanges, are an important part of those discussions. Most commonly used order types work in a relatively straightforward manner. But sophisticated investors, including high-frequency traders, often use complex order types to compete over fleeting opportunities in ultra-fast modern markets.

Direct Edge introduced several new order types in 2009 after conferring with two high-frequency trading firms but didn’t properly disclose to the public how they worked, according to the SEC order.

Bats, based in Lenexa, Kansas, entered into a settlement to “put this matter behind us,” it said in a statement.

The previous record for a settlement with the SEC was held by Nasdaq OMX, which agreed to pay a $10 million fine for its mishandling of the initial public offering of Facebook Inc. in 2013.

Related

The SEC also closed a separate investigation into order types at the Bats exchanges with “no action taken,” according to the Bats statement and a letter from the SEC posted on Bats’ website.

Bats was aware of the SEC’s investigation into Direct Edge before the merger of the two companies, and had set aside money to eventually settle the case, according to people familiar with the matter. But after the merger, executives learned that the fine would be higher than they initially expected and that former Direct Edge chief executive William O’Brien ’s difficult relationship with the SEC had affected the discussions with the commission, they said.

That was a key reason O’Brien was forced to resign as president in July, the executives said.

“While I will not comment on the details regarding my departure from Bats, any suggestion that I was unhelpful – deliberately or otherwise – regarding an SEC investigation is completely without merit,” O’Brien said.

A Bats spokesman said “there’s no evidence that Bill O’Brien’s relationship with the SEC had any negative impact on the case.”

The investigation into Direct Edge order types was spurred by a former high-frequency trader, Haim Bodek, who filed a whistleblower complaint with the SEC in 2012. The Wall Street Journal reported in a page-one article in September 2012 that the SEC had launched an investigation into order types after his 2011 complaint.

Bodek said at the time that certain order types, such as one offered by Direct Edge called Hide Not Slide, gave some investors an advantage over others.

In an interview Monday, Bodek said the settlement agreement was “the first official vindication” of the work he had done in trying to expose issues with order types at exchanges. He is eligible for a whistleblower award from the SEC and will apply for one, according to Shayne Stevenson, Bodek’s lawyer and head of the whistleblower practice at Hagens Berman.

Bodek said that the most revealing aspect of the SEC order was the role of two unnamed high-frequency traders, referred to as Trading Firm A and Trading Firm B, who were described as influencing the way Direct Edge created some of its new order types.

Trading Firm A told Direct Edge that by introducing a certain order type it would increase the number of orders it sent to Direct Edge to between 12 million and 15 million a day, from between four and five million a day, the SEC order said.