The five young men with British passports were clearly jihadists, on foot at the border crossing from Turkey into Syria. Their beards were very long, and trousers very short – a common fashion among Sunni Salafist believers – and the war was one of the very few reasons to be there.

The Turkish immigration officer did not bat an eye, clearly used to seeing such travelers. He stamped their passports as casually as he stamped a reporter’s American one.

That was October 2012, when Turkey was in the midst of a years-long, live-and-let-live policy to allow jihadists and rebels of all kinds free passage to Syria in a bid to hasten the fall of President Bashar al-Assad.

But today, predictions that Turkey would suffer blowback from that policy are coming true, with Syria’s violence – and more than 2 million refugees – spilling back across the border, as the war grinds through its fifth year.

Many of those rebels who flocked to Syria have now joined Al Qaeda-linked groups or the self-described Islamic State (IS), and shown they are ready to strike against their own enemies, such as ethnic Kurds, inside Turkey.

Analysts say Turkey’s response has been soft on IS, even as the jihadist group has successfully recruited Turkish citizens to its cause, while being very hard on militants of the Kurdistan Worker’s Party (PKK), who operate within its borders and from bases in northern Iraq. The state appears to view the PKK, which it has fought for most of the last 40 years, as much more of a strategic threat. Yet IS has proved itself increasingly costly for Turkey, first as a point of discord with its strategic allies and now as a real threat to its internal security.

“Turkey is waking up very belatedly to the threat of these extreme Salafi jihadists inside Turkey,” says Mustafa Akyol, an author and columnist for Hürriyet Daily News in Istanbul.

Recent shifts on Islamic State

The Oct. 10 double suicide attack in the heart of the Turkish capital, Ankara, which killed more than 100 people at a peace rally, is the most potent example of how Syria’s war is affecting Turkey’s citizens and its politics.

But it is just the latest one: Three bomb attacks since June have all been carried out by radicalized Turkish citizens linked to IS, each one targeting pro-Kurdish groups. They are an extension, analysts say, of the fierce battle between IS and Kurdish fighters across northern Syria and Iraq.

In a change of heart in July, after a bombing killed 33 people in the border town of Suruç, Turkey agreed to allow US planes to strike IS inside Syria from its Incirlik airbase. Those began in mid-August. Turkey also has clamped down on border crossings, and arrested scores of wannabe foreign fighters on their way to Syria.

But so far Turkey has launched only a handful of airstrikes against IS positions – while hammering the PKK in northern Iraq with hundreds of sorties that officials say have killed 2,000 militants. That surge, along with PKK attacks that have killed scores of Turkish policemen and soldiers, has effectively ended a years-long Kurdish peace process.

Mr. Akyol, author of “Islam without extremes: A Muslim case for liberty,” describes the state establishment’s mindset as likely perceiving the PKK as “the real, long strategic threat.”

Focus on regime change in Syria

“Why? How many followers would IS have in Turkey, a couple of hundred, maybe a couple of thousand? It’s not a mass movement, and they probably think … the US will bomb, and IS will disappear,” says Akyol. “Whereas the PKK, with millions of followers, is something that really threatens Turkey’s territorial integrity maybe in the future. So that’s why they may be putting all their attention on the PKK.”

That is one result of a “90-year tradition of seeing the Kurdish separatists as your main problem,” he says.

But the spillover of the Syrian war is also a result of Turkey’s miscalculations and obsession with toppling Mr. Assad, says Fadi Hakura, a Turkey analyst with Chatham House in London. When Syria’s popular uprising began in early 2011, Turkey’s leaders, including then-Prime Minister Recep Tayyip Erdoğan, assumed along with many others that the Syrian president would be overthrown quickly, similar to the removal of Hosni Mubarak in Egypt.

Turkey’s Syria policy focused on regime change in Damascus, says Mr. Hakura, while the potential for future blowback was underestimated.

“But it was quite clear as the Syrian quagmire dragged on, and the regime in Damascus demonstrated its resilience, that Turkey [had] entered into trench warfare, and the spillover of violence was, I think, quite inevitable,” says Hakura.

Could IS shift targets?

“We see that the frequency [of attacks in Turkey] is growing, the level of violence is intensifying, therefore the likelihood this trend could grow clearly exists,” he says.

The bombings are not expected to change the results of snap elections on Nov. 1, in which a pro-Kurdish party is likely to maintain its 13 percent support and again deprive President Erdoğan’s Justice and Development Party (AKP) of its majority, as it did in June parliamentary elections.

So far IS has not struck any Turkish state or AKP targets, instead focusing on what it portrays in Turkish language pamphlets as the “atheist gang” of godless: secular Kurds. But the group, which considers Al Qaeda too soft, also has strong words against Turkey – even though for years the nation has been the lifeline for its fighters in Syria.

“They are not in favor of AKP, they actually accuse Erdoğan of being a puppet of the West, or a hypocrite or infidel,” says Akyol. “But they have been particularly angry at Turkey’s opening of the Incirlik base to the US, which for them was Turkey joining the ranks of the Crusaders.”