TOKYO—Nintendo Co. said its operating loss nearly doubled in the most recent quarter from a year earlier, deepening doubts about the Japanese videogame maker's ability to reverse its tumbling fortunes.

Nintendo has been counting on strong sales of popular titles to pull it out of a slump that has resulted in three straight years of operating losses. Yet even though racing game "Mario Kart 8" performed well, those results...