Speaking to investors and analysts at the Sandler O'Neill Global Exchange and Brokerage Conference on Thursday, Joyce said Nasdaq has to "go back to the drawing board and come up with something sensible."

On Wednesday, Nasdaq disclosed the highly anticipated compensation plan, saying it would pay $13.7 million in cash to member firms that suffered losses, including the $10.7 million profit that Nasdaq made from first-day trading in Facebook and the maximum $3 million allowed under its rules to make good for botched trades.

The rest of the payments--which received sharp criticism from rival NYSE Euronext
NYX, -0.52%
-- would come from trading discounts, which Nasdaq said would cover the Facebook losses within six months "for the vast majority of firms."

Reached for comment on the sidelines of the conference, Joyce said he believed trading firms would "use whatever avenues at their disposal, including legal means" to be reimbursed for the losses.

During his presentation, Joyce said the roughly "$27 million of so-called credit trades is underwhelming at best," adding that the Nasdaq needs "to go back to the drawing board and prove they are a client-focused organization."

Knight Capital is one of the four major U.S. wholesale market makers--firms that execute orders for individual investors supplied by the investors' brokerages.

That group, which includes UBS AG (UBS, UBSN.VX), Citigroup Inc.
C, -1.64%
and Citadel LLC, is estimated to have lost more than $100 million collectively related to bad Facebook trades, according to people familiar with situation. Some estimates had that loss total approaching $200 million.

Knight Capital itself has said it may suffer losses of up to $35 million linked to its trading in Facebook's IPO.

Commenting on the collective-loss estimate, Joyce said, "I have no doubt that (the figures) are true, and I would not at all be surprised if the loss number was at the higher end of the range."

Intraday Data provided by SIX Financial Information and subject to terms of use.
Historical and current end-of-day data provided by SIX Financial Information.
All quotes are in local exchange time. Real-time last sale data for U.S. stock quotes reflect trades reported through Nasdaq only.
Intraday data delayed at least 15 minutes or per exchange requirements.