VIENNA—A top State Department official discussed the crisis gripping Iraq with Iranian diplomats, a senior U.S. official said late Monday, the most direct talks yet between Tehran and Washington on the issue.

The U.S.-Iranian meeting took place alongside international nuclear negotiations in Vienna, officials said, in a development that emerged as an unexpected outgrowth of the crisis in Iraq, where a radical Sunni militia has...