Frank Troise is three weeks into a job as chief executive of Investment Technology Group Inc. and he faces a familiar task—plotting a new path forward for the brokerage firm.

The company he first joined almost two decades ago has been under pressure since last summer when executives said they were working with the Securities and Exchange Commission on a settlement related to ITG’s “dark pool” and a pilot trading program. The...