A software glitch knocked out trading on the largest U.S. options exchange for more than three hours Thursday, raising fresh questions about the reliability of critical elements of the financial market.

The Chicago Board Options Exchange was forced to delay trading, including in its heavily traded stock-index options contracts, until early afternoon, leaving traders and brokers in its Chicago pits to field angry calls from...