CANBERRA—Australia's deputy finance minister stepped down from his role amid a corruption probe that had threatened to engulf Prime Minister Tony Abbott in his first major political storm—dramatically reversing a pledge only hours earlier to "soldier on."

Arthur Sinodinos' decision to step aside while the investigation into his business activities continued provided relief to the government of Mr. Abbott, who as recently as Tuesday said he had full confidence in the assistant treasurer. Mr. Sinodinos announced...