In search of his country’s Tsarist roots, Vladimir Putin has steered Russia on a course towards unforgiving political authoritarianism, buttressed by state capitalism and legitimised by orthodox religiosity and patriotism.

Towards the West, today’s Russia appears prickly, often aggressive, always obstinate and pig-headed. But in the years following the Cold War, Britain and its allies vigorously encouraged Russia to join the Western alliance of democracies. Hopes and expectations ran high. Putin – like Yeltsin and Gorbachev – saw this as desirable too, but only if, in geopolitical terms, Russia could join on an equal basis with the United States.

This was, of course, utterly unrealistic, out of kilter with the new balance of power. But it was a conviction as deeply embedded in the Kremlin’s psychology, as it was poorly understood in the West. Moscow was blinded by a vision of its past, the empire that, having defeated Hitler, enjoyed world power status. It felt snubbed. Western diplomacy was blinded by a vision of the future, the global triumph of liberalism. It did not get what Putin, the “spoiler”, was playing at.

For years this tension defined relations with Russia, right through the Obama reset in 2008. This proved a strange diplomatic dance, neither side quite understanding the other, and consequently stepping on the other’s toes. Both sides persisted in the embrace, albeit with evaporating conviction – until the cord finally snapped, as Putin himself had warned would happen over Ukraine.

When it did happen, the cut was deep, severe and emotional. It froze Russia out of cooperation with the West. Putin was ousted from the G8 and other summits. Ties with NATO were suspended, economic sanctions imposed, the Cold War doctrine of deterrence dug up, dusted off, and redeployed on both sides.

This unequivocal response to the annexation of Crimea was necessary and inevitable. Russia blatantly flouted international law in Ukraine, showing utter disregard for territorial borders. It is indisputably correct that our Eastern European allies now ask and receive NATO reassurances, in the form of conventional military deterrence.

However, when the Prime Minister meets his NATO colleagues in my ancestral home of Warsaw, in a month’s time, the real question is whether we should want deterrence and containment to become the new normalcy, suspending our relations with Russia indefinitely. The dramatic torrent of events in the Middle East and North Africa appears to be dragging us another way. Already, the United States and Russia are – quietly and uneasily – working together in Syria in ways unthinkable during the Cold War. In spite of the different objectives, there are common strategic interests that pull us together.

In certain diplomatic and defence circles in Europe this may be unspeakable, but it will have to be said: at some point Britain and its allies will need to start thinking about constructing a new Russia relationship, a more realistic and durable alternative to our post Cold War dance with Russia. And in the face of the political chaos on Europe’s southern borders, and also Russia’s southern borders, that moment may come sooner rather than later.

Putin is not going to go away soon, and even if he did, his departure is unlikely to awaken a rush of desire to embrace Western style institutions. While such institutions have support in Russia, it would be a catastrophic misreading of Russia to assume this would, following Putin’s demise, lead – Maidan-style – to Russia’s own Revolution of Dignity. It seems just as likely, if not more, that, following a prolonged period of disorder, revanchist patriotism would emerge triumphant.

Freezing out Putin altogether does not serve Britain’s strategic interests, nor should we limit ourselves to tit-for-tat deterrence strategies. Russia’s obsession with NATO and political encirclement is strong as ever. As sure as night follows day, Putin will meet our new deterrents every step of the way, until he can no longer keep up and, feeling cornered like cat, lashes out in what threatens to become a self fulfilling prophecy.

We need to find novel ways of working together with Russia, re-establishing a modicum of trust. This cannot be another “reset”, nor can it lead to the rehabilitation of Russia inside the Western liberal order. Russia has placed itself outside of that order. We need to start from the assumption that for now it is going to stay there. Reviving greater expectations would be to lapse in the mistake of post Cold War idealism.

Yet this still does not make Russia Britain’s perpetual enemy. Disruptive as it may be, Putin’s Russia is not our ideological nemesis, to be defeated like the Soviet Union. We need to learn to look at Russia through the cold but clear prism of our interests, just as Russia’s foreign policy establishment undoubtedly looks at us. Viewed through this prism, our ties with foreign states are a matter of expediency, not sentimentality, or the projection of the liberal ideal. In the nineteenth century, this was the common view. As Lord Palmerston put it, “We have no eternal allies, and we have no perpetual enemies. Our interests are eternal and perpetual, and those interests it is our duty to follow”.

Russia will be our adversary. Russia will be our partner. Mostly it will be both at the same time. This is the complexity of diplomacy, and the balance of power. It is a complexity that Putin, steeped in realist logic, will understand well. So should we.