Show captionNorthern Ireland football fans. ‘We might wish for a world in which more of Northern Ireland’s people shared a collective identity, but that is not is the world we live in.’ Photograph: Niall Carson/PA

Who knows if Karen Bradley drinks Scotch, but I hope she doesn’t think Northern Ireland is awful after her first visit as secretary of state. Because it isn’t; it has transformed itself over the years. But politically it remains bloody difficult, as the prime minister once liked describing herself.

Theresa May promised to be bloody difficult in Brexit negotiations, but it is Northern Ireland that is delivering on that pledge. This corner of western Europe has provided a never-ending essay question that has distracted British ministers for generations. And it is now at the centre of the toughest, most convoluted test the British state has ever set itself.

It was always going to be so, but practically the entire UK political class failed to grasp why. Working on Brexit communications in Downing Street, I became somewhat repetitive on the subject.

As Bradley will discover, Brexit has unsettled one of the most intangible but important features of the fraying Northern Ireland settlement: the ability of its citizens to imagine themselves into different nationalities. This is why the border question is so difficult: it is about psychology as much as the practical mechanics of border controls. How does anyone know what nationality they are? Do they belong to the country to which they pay taxes, or whose football team they support?

Some weeks before the UK and EU agreed terms on the withdrawal phase of Brexit – including the Irish border – two football teams with Ireland in their name were agreeing terms on their own withdrawal, in this case from World Cup qualification. The island of Ireland fields two international soccer teams, one from each jurisdiction. Then it gets complicated. The Northern Ireland team, once perceived as exclusively unionist in support, is now much more inclusive. But it would be wishful thinking to claim every football fan in Northern Ireland supports this team. Many people support the Republic or, as some prefer, they simply support Ireland.

It should be no surprise that people who identify as Irish opt to support a team named Ireland, and embrace its symbols – the flag and national anthem. Many fans of Northern Ireland (though by no means all) display symbols of Britishness. The union flag is waved. God Save the Queen is played before games.

We might wish for a world in which more of Northern Ireland’s people shared a collective identity, but that is not is the world we live in. Nations are imagined communities, to use an old truism. The people of Northern Ireland have, over time, constructed separate psychological spaces for their identities. And part of the reason for enduring political instability is that neither monolithic identity can win. Both are inherently insecure.

People who feel Irish live in the island of Ireland, but not the state called Ireland. People who feel British live in the British state, but not on the island of Great Britain.

The Good Friday agreement was elaborately engineered to reflect this. It not only instituted power-sharing, but created a legally enforceable right to identify as British, Irish or both. The agreement is fastidious in keeping Northern Ireland within the UK until a majority votes otherwise. But it is expansive when describing the right of people there to be part of the “Irish nation”. To make people who feel Irish relaxed about Ireland being partitioned as a matter of legal fact, the agreement sought to soften the border in people’s minds: to help them imagine it wasn’t there.

What many in London did not anticipate was how intensely the Irish government would argue that the UK’s decision to leave both the EU’s single market and the customs union would, in practical effect, fetter the ability of people in Northern Ireland to imagine themselves in an Irish nation. Dublin maintained that divergent rules on either side of the border would render much of the north-south cooperation envisaged in the Good Friday deal impossible – disrupting the psychological effect of that agreement.

It now believes the text agreed in December, which provides a “backstop option” of “full regulatory alignment”, is the first step in preserving meaningful all-Ireland cooperation post Brexit. There is as yet no explanation for how full alignment can be achieved while the UK takes control of its laws, its borders and its money.

The quickest solution, some sort of economic border between Britain and Northern Ireland, is anathema to the Democratic Unionist party, which keeps the government in power. The DUP argues, not illogically, that if barriers between Northern Ireland and the Republic undermine Irish identity, barriers between Northern Ireland and the rest of the UK must threaten British identity. The prime minister herself acknowledged as much in a letter of guarantee to Northern Ireland’s unionists.

Talk of imagination will strike some people as overwrought. So too will promises of blue passports. And Brexit stamps. But as with every new Northern Ireland secretary, Karen Bradley will learn that the psychology of imagined identity is a powerful thing. The prime minister has an even harder task: to contrive a Brexit that satisfies the imaginations of not just the fragmented Northern Irish, but also of those in her party with their own psychological preoccupation: total liberation from Europe. As with the hardest essay questions, there is no right answer but many wrong ones.