ISTANBUL—The rapid advance of Islamist militants across Iraq and Syria in recent days spotlights how Turkey, a North Atlantic Treaty Organization member, is acutely exposed to violence engulfing its southern neighbors.

On Thursday, Ankara was engaged in an intense effort to free 80 Turkish nationals captured by the Sunni militant group Islamic State in Iraq and al-Sham (ISIS), an al Qaeda offshoot, in the Iraqi city of Mosul....