When the clock struck 10 last Thursday night, there was a moment of collective disorientation. With each tolling of the bell, the solid political ground we had been standing on was shaken by tectonic shifts below. On television, the anchors sounded unconvinced by the news they were announcing: according to the exit poll, the Tories had lost their majority and Labour had gained seats. “Boy, oh boy, oh boy,” David Dimbleby said on the BBC, “are we going to be hung, drawn and quartered if this is all wrong!”

For weeks the polls had told us this was highly unlikely: most were predicting a Tory victory somewhere between comfortable and landslide. And for two years before that, journalists and pundits had told us it was not possible; the only logical conclusion for a Labour party led by Jeremy Corbyn was a ruinous election defeat. This was not simply a partisan view, but one broadly shared across the entire spectrum of mainstream politics.

Three days before the vote, the party activist blog Labour Uncut quoted one campaigner who had just returned from the North East predicting a “nuclear winter for Labour”. The day before, a senior member of Corbyn’s team told me that he assumed the Tories would achieve a double-digit majority. The narrow possibility of a hung parliament was out there – but there were far more guesses of a triple-digit wipeout.

The campaign itself had its surprising moments: leaked manifestos, sudden U-turns, fields of wheat and absolute boys, and a No 1 record branding the prime minister a liar. But the outcome was never in doubt.

Twelve months earlier, during the EU referendum campaign, the political class had profoundly misjudged the national mood, and plunged the country into chaos. This election was supposed to provide a correction.

Then came those bells, and the unravelling of all our assumptions in real time. There is a distinction between witnessing something one is told is unlikely to happen and something one is told cannot happen. The former is a surprise, a challenge to our understanding of how things work. But the latter is a shock, and it forces us to reckon with the question of whether things are working at all. As the results came in overnight, with huge swings to Labour in seats that the Conservatives had targeted, and gains in places where Labour was not supposed to be competitive, each new upset seemed to rewrite the rules by which we understood electoral politics operated. By dawn, the whole rulebook had been shredded. Throughout the night, panels of pundits who had told us with great confidence that this could never happen were telling us with equal certainty what would happen next.

Electorally, the night was confusing. As the votes were being counted, nobody had a clue how the night was going to pan out. Fifty-two seats were returned with majorities below 1,000 votes, including eleven with majorities of less than 100. On those narrow threads hung our future. And now the counting is done, we’re still not sure.

Politically, the result was much clearer. The party that came second had emerged resurgent, while the party that came first was humiliated. Theresa May’s days as party leader are numbered, while Jeremy Corbyn’s position has been unexpectedly secured. The day after the election, May returned from Buckingham Palace to Downing Street and announced that she would do a deal with Northern Ireland’s Democratic Unionist Party. “What the country needs more than ever is certainty … Now let’s get to work,” she said, as if the idea to stop work and have an election had belonged to someone other than herself. She sacked her advisers. Meanwhile, the Tories have rallied around her as though nothing had happened. But it had. We all saw it. It wasn’t pretty.

I spent election night in Harrow West, the constituency in north-west London I’d been covering throughout the campaign. It’s a seat that had always been Conservative until 1997, when Labour overturned a Tory majority of 17,800 for a narrow win. Twenty years and four elections later, Labour was still clinging on, with a slender majority of 2,208. It was No 19 on the Tories’ target list. “With the exception of 2001, it’s always been closely fought,” the local MP, Gareth Thomas, told me last month. “I try to focus on the basics of being a good constituency MP. I think that has a lot to do with it. But I think we have been lucky. And my luck may be about to run out.”

Nothing was being left to chance. Throughout the month, I followed Labour canvassers as they knocked on doors, held rallies in parks and campaigned against education cuts.

When I arrived at Harrow Leisure Centre at 2.30am on election night, local Labour activists were cheering at the television. What they thought would be a wake had ended up a shindig. It was clear by that stage that Thomas had netted a healthy victory. Almost an hour later, at 3.20am, he was declared the winner, with a 13,314 majority – doubling his previous highest margin of victory.

It was beyond unexpected: it was unfathomable. The local Labour party had done its sums and tallied its returns. It had canvassed, pursued and chased up all its known voters. It did not see these new voters coming. They were not on its radar. They had noticed some surprisingly long queues of young people at the polling station by Rayners Lane, but that didn’t explain it all. The party won Tory wards, and assumed that it picked up some Green switchers. Everyone was immensely grateful for them. But they had no idea who they were.

This was the story of the election, and it is the story of this political moment. When Big Ben called time on Thursday night, we saw clear evidence of a political realignment that the media and the political establishment had dismissed with hostility, and now regarded with confusion. We saw a polity that has lost touch with its people; a political culture unmoored from the electorate, and a mainstream media that drifted along with it. The election did not create that dislocation; it was merely the clearest and least deniable manifestation of it so far. We are in new territory. And we don’t know where we’re going.

“The only feeling that anyone can have about an event he does not experience is the feeling aroused by his mental image of that event,” the American political commentator Walter Lippmann wrote in his landmark 1922 book Public Opinion. “That is why, until we know what others think they know, we cannot truly understand their acts … Our opinions cover a bigger space, a longer reach of time, a greater number of things, than we can directly observe. They have, therefore, to be pieced together out of what others have reported.”

Our views about the world do not come from nowhere. We begin with preconceptions and personal experiences, and then adjust our opinions on the basis of what we see, know, hear and feel. But those influences are not neutral. They bend to the prevailing winds of power and status, because we pay more attention to sources that possess greater authority. When it comes to political matters, we are more likely to listen to politicians or newspaper columnists than people we talk to on the bus or in the supermarket, because we assume they are more informed about the relevant facts.

There were good reasons to doubt that the election result would turn out the way it did. Labour started more than 20 points behind, by some estimates, after two years of infighting and disarray. Last month, the party was heavily defeated in local elections. The carnage and heartbreak of terror attacks in Manchester and London had stalled the campaign and given May the chance to play the role of prime minister.

But it is also true that the reported predictions of a Tory landslide – and the premises on which they were based – were so dominant and pervasive that people chose to filter what they saw through that lens. These assumptions coloured observations about the election campaign, which were then either dismissed or adjusted to align more closely to the original predictions.

What we saw did not chime with what we thought we knew. But what we thought we knew was underpinned by a political calculus that no longer held.

So when mainstream commentators saw the huge crowds that turned out to see Corbyn speak, they were keen to explain that this did not necessarily mean huge electoral support – which is correct – but they rarely made much effort to think about what it did mean. When they heard reports of masses of young people registering to vote, they were also keen to point out that many were concentrated in places that would be unlikely to affect the outcome of the election (which is, again, correct) and that, in any case, young people could not be relied upon to actually vote (also true).

When they saw the trend of hard-left parties surging and even, at times, eclipsing the soft left across Europe – in France, the Netherlands, Greece, Spain, Portugal – they said Britain was different, and that even if something similar were to happen here, its effects would be blunted by the first-past-the-post system.

On the day of the vote, Labour MPs in safe seats said things were going well in their own constituencies, but that they were hearing terrible things from the marginals. Others were saying that Labour turnout was surging, but only in places where the party already had huge majorities. The pessimism had become so intense and paranoid that one Labour MP told me last year, during the London mayoral election, that the party was only leading because the Tories were deliberately throwing that contest. That way, the MP suggested in all seriousness, Corbyn would be less vulnerable to a leadership challenge, which would mean that the Tories could reign supreme on a national level.

There were signs. Not of the outcome we have witnessed, necessarily, but of a result that defied the predictions of abject calamity. It all depended on who you spoke to. But the trouble is that in a moment of flux like this, the people you would usually expect to know may well have no idea. Labour MPs, many of whom had already shown themselves to be out of touch with the views of their own party’s members and supporters, may not have been the best judges of the national mood. Canvassers, who tend to focus on people whose preferences are already known, would be unlikely to pick up a surge of new voters.

As long as journalists, politicians and party workers were – for the most part – talking to each other, they were only reinforcing each others’ narratives. If events had been unfolding as normal, then everything they said would have made sense. But nothing was normal. And as long as they were tied to the old way of thinking, there was no way for them to know it until the votes came in.

The day before the election, the American polling analyst Nate Silver spelled out three possible scenarios for the UK election: a hung parliament, a narrow Tory majority, and a Tory landslide. Because “UK polls have not historically been very accurate,” he wrote, each of these three outcomes was equally likely – but nobody in Britain was claiming this.

A few days earlier, Silver warned of what he called “the First Rule of Polling Errors”: “When the conventional wisdom tries to outguess the polls, it almost always guesses in the wrong direction.” In other words, when pundits declare that the polls are overestimating one side, they are probably actually overestimating the other side. This had been the case with Brexit and the election of Donald Trump. Now, for the third time in less than a year, the commentariat was at it again.

But while it was possible to see how most voters had formed their first impressions of Corbyn and May from the image presented by the media, what became clear to me while I was covering the campaign was that the impact of Fleet Street was not decisive. Thanks to the proliferation of online media sources, the decline in newspaper readership and weakening loyalties to established brands, the press does not have the same electoral clout it once did. (According to YouGov, more than half of Sun readers didn’t vote.)

The night the Labour manifesto was released, I went to a hotel in Wembley, close to Harrow, where a focus group of undecided voters, brought together by research company Britain Thinks, intimated that they were warming to Corbyn, in part because they preferred him, as a person, to the prime minister. Some had serious reservations – calling him “unfocused”, “untidy” and “lacking personality” – but most described him as “a man of principles”, an “idealist”, “honest” and “underestimated”.

They saw Theresa May as “strong” and “clever”, but also “uncaring”, “untrustworthy” and “unpredictable”. One voter said: “He’d buy a round, Theresa wouldn’t.” The group were asked if they would trust May to look after their house while they were on holiday. The consensus: “The house, yes. But not the pets.”

During the EU referendum, much of what was wrong with Britain was blamed on foreigners – either the faceless bureaucrats in Brussels who took our money at the expense of the NHS, or immigrants, who, it was claimed, took our jobs and plundered our benefits. But this time round, there was no one else to blame. There was a concern in that room in Wembley that Britain had become too harsh and unforgiving. One woman said she thought things had swung too far the wrong way, and that it was time to “make things fairer”. Another agreed. “We need to show people we care about them,” she said.

If you spend a day in the Harrow Law Centre or a morning in the local branch of Citizens Advice, you get a clear picture of what that unfairness looks like, and where it comes from – a toxic cocktail of low pay, scarce housing and spending cuts. In Harrow, 37% of employees are working in low-paid jobs, and 42% of jobs in the local area pay less than the London living wage. In 2015 the borough was named, along with West Somerset, as the worst place for low pay in the country. Meanwhile, housing costs are high and rising: tenants in the borough spend 61% of their earnings on rent. Last year, on average, twice every day Harrow’s food bank gave out three-day emergency supplies for children.

“We’ve always seen a lot of people come through struggling with debt,” said Susan Kearney, a supervisor at Citizens Advice Harrow. “But it used to be people with credit cards. Now it’s rent arrears, council tax, gas, electric – essentials, basically.”

After the 2010 election, the Conservatives insisted on a period of austerity, claiming that it was necessary to repair public finances in the wake of the global banking crisis. The poor and the public sector have borne the brunt of these cuts – but after seven years, the pain of austerity has spread well beyond the very poorest.

“We’ve seen teachers and nurses struggle with housing problems in a way that we would never have done in the past,” said Pamela Fitzpatrick, a Labour councillor who runs Harrow Law Centre. “They just can’t survive in London on those salaries. It all comes down to a lack of security and stability.”

As this sense of precariousness broadened to touch those who had never felt it before, and the desperation felt by an ever-widening cross-section of society deepened even further, we should not be surprised that there was an electoral backlash.

A fortnight before the election, I met Mohammed Ameripour, who lives with a severe cerebellar disorder that affects his speech and vision, at the law centre. He was declared fit for work, even though he has a full-time carer, and ruled to have no problem with mobility, even though he has to use a wheelchair. The benefits he was reliant on for 19 years were cut. “All the doors were shut to me,” he said. “Every door. I couldn’t get them to help me.” Thanks to the law centre, he won his case on appeal. “Some people are claiming benefits when they are healthy, and that’s not right,” he said.

“I think Jeremy Corbyn is weak. I prefer Theresa May. I think she is a leader. But I don’t like what she has done to benefits. I don’t like what she has done to me.” He was carrying his postal vote for Corbyn with him the day we spoke.

Gareth Thomas, the MP for Harrow West, was one of several Labour parliamentarians who had no intention of voting for Corbyn in the 2015 leadership race, but nominated him so that he would get on the ballot and ensure a broad debate within the party. None of the MPs who lent Corbyn their nominations ever imagined he would win. Nor did he.

For Corbyn, the accidental candidate, the leadership contest was a blur. “Everything sort of took off,” he recalled when I interviewed him a year later. “We didn’t have a campaign. We didn’t have an organisation. We didn’t have any money. All we had was my credit card. That lasted about a week. Then we started raising money … Time for reflection was very limited, because from the moment I was nominated I was on a train.”

But Corbyn began to pull in crowds, and his polling numbers started to rise. Lots of new people joined the party to support him. His campaign posed a direct challenge to the orthodoxy of the previous three decades, under which the Labour party was an electoral machine run from the top down, with the insistence that it could only win elections with ruthless discipline and tabloid-friendly appeals to the centre ground.

After Labour’s defeat in the 2015 election, under the leadership of Ed Miliband, the Fabian Society published a document titled The Mountain to Climb, which argued that four out of every five voters Labour would need to win over in order to return to power would be former Tory supporters. When the website LabourList asked Corbyn how he would respond to this challenge, he proposed a very different strategy. “I think their approach to the research is from the wrong end of the telescope,” he said.

“[There are] young people who didn’t register, who didn’t vote,” he continued. “Those that did vote were overwhelmingly Labour, so I think there’s a whole area there, and this [leadership] campaign is demonstrating that. Secondly, [there were] reliable Labour voters who disappeared into the arms of Ukip, or not voting, because they didn’t feel the Labour party represented anything they wanted to hear. I think we can grow our support that way. Do we have to win back people who voted for other parties? Yeah, but we have to say to people, in a very clear way, what we’re offering.”

That was the proposition: to expand the electorate, broaden Labour’s coalition, and reach out to disaffected voters with a new and more radical offer. Many people thought this would not work, but a surprisingly large number insisted it wasn’t even possible.

The Fabian Society was unconvinced. Corbyn’s numbers, it insisted, did not stack up. “The Fabian analysis has looked at Labour’s electoral mountain to climb from both ends of the telescope,” it responded. “Any strategy to win the next election will require Labour winning over a lot of people who voted Tory in 2015. That is a fact all the leadership candidates must come to terms with.”

These were, arguably, going to be existential questions for Labour, however they were answered. Many claimed that the party risked extinction under Corbyn, while largely ignoring the fact that, after losing two consecutive elections with moderate leadership and policies, it might have been heading for extinction anyway.

In the end, one in 10 Tories and one in five Ukip voters switched to Labour. Young people turned out in greater numbers and were more likely to vote Labour than in previous years. Labour won 40% of the total national vote and gained 30 seats, yet it remains 58 seats behind the Tories.

This result will not settle the longstanding debate in the Labour party about whether the views of members should take priority over those of MPs, or the related matter of whether concessions to the centre are necessary to win elections. But, for now, it should at least establish that these are open questions and that the understanding of what makes a party electable is not a fixed science, but contingent on context, place and many other external factors.

Labour supporters at the Harrow count on election night. Photograph: Graeme Robertson for the Guardian

There has been a great deal of contrition from Labour party backbenchers lately, the most vivid illustration of which was the hero’s welcome Corbyn received when entering parliament earlier this week. The party is more inclined towards unity than it has been for quite some time. Given everything the parliamentary Labour party (PLP) has said and done over the past couple of years, and what Corbyn has managed to deliver all the same, that is to be expected. What remains to be seen is whether those MPs are willing to understand that they didn’t just get Corbyn wrong or this election wrong – they completely misunderstood the politics of contemporary Britain.

Corbyn emerged in the wake of a global financial crisis, in a country rocked by the phone hacking scandal, the MPs’ expenses scandal and Operation Yewtree. His ascendancy represents a desire for a more participatory, bottom-up kind of politics that takes on not only the Tories in parliament, but inequality in the economy, unfairness in society and power where it has not previously been held to account.

The fulfilment of that desire will mean less power for MPs and more for party members, and a politics guided as much by the needs of the electorate as it is by the concerns of politicians. In short, it demands both an ideological and methodological shift from the orthodoxies of the past two decades. The PLP did not understand this from Corbyn’s first party leadership victory. It didn’t even understand this after he defeated an attempted leadership coup with an overwhelming mandate. We should not assume those lessons have been learned now.

When Harrow West MP Gareth Thomas went to the count in 1997 – the night he was first elected – he had prepared one speech to deliver if he came second and another in case he came third. He had not written a winner’s speech, because he didn’t think he needed to. When he took to the podium on Thursday, he paid tribute to Corbyn as “the only leader to emerge from this campaign with any credit”. One wonders whether that was another impromptu speech he hadn’t reckoned on making.

In the framing of the campaign from the outset, the Tories were the predators, encroaching on Labour territory. Labour was stuck in a defensive crouch, hoping to protect what it could. The question was not whether the party would lose seats, but how many. This was the story we assumed we were covering.

When you arrive at Harrow & Wealdstone tube station, you can turn left into Harrow West, or right into Harrow East. I always turned left: Harrow West was 19th on the Tories’ target list. Harrow East, on the other hand, was a Tory seat with a comfortable majority of 4,757. It would have been 45th on Labour’s list of target seats if anyone thought Labour had target seats in this election.

By the time I turned up to the count, it felt like the real story had been in Harrow East all along – only nobody knew it, including the people campaigning there. The announcement of the results had been delayed, it turned out, because the result in Harrow East was so close. “We had no idea,” one Labour canvasser told me. Bob Blackman, the Tory MP, could be seen nervously pacing the floor of Harrow Leisure Centre – where the results for both local seats would be announced – until he learned he had scraped by with a reduced majority of 1,757 – a 5% swing to Labour.

We have been so focused of late on the problems of the centre left that the crisis in the mainstream right has only fleetingly disturbed our gaze. It has been easy to forget that what brought Britain to this point was weakness within the Conservative party. The Brexit referendum was concocted by David Cameron to quell internal Tory strife and shore up the party’s right flank against Ukip. When that gamble failed, Cameron fled – and May became prime minister after her two biggest rivals stabbed one another in the back.

May called this election because she could not trust her own MPs to back her through the Brexit negotiations. Now her gamble has failed – but she has stayed on as prime minister in any case. Labour’s crisis was seen as the internal pathology of a divided left. The Tories, however, have made a habit of imposing their crises on the nation.

May ostensibly called the election to strengthen her hand in Europe, but then said nothing about what her negotiating strategy was going to be. Her pitch was entirely performative (“I’m a bloody difficult woman”) and epigrammatic (“Brexit means Brexit”; “A bad deal is better than no deal”). But not even remotely substantive. Difficult about what? What does “hard Brexit” mean? What would no deal look like?

Despite all its shortcomings, for a while this looked like a strategy that might work for a domestic audience. The decision to leave the European Union was not made on the basis of details, but the broad strokes of melancholic nationalism, exclusion, anomie and sovereignty. In Corbyn, May faced an opponent who had struggled to negotiate with his own party and whose patriotic credentials were in doubt – not only did he not sing the national anthem, he had refused to say that he would drop a nuclear bomb on anybody.

May’s electoral appeal as the nation’s tough negotiator-in-chief might have held strong were it not for two problems. The first was her glaring weakness as a candidate. Backing away from her own manifesto and staying away from debates made the prime minister look shifty, weak, and lacking in resolve – precisely the kind of person you do not want arguing your case in foreign parts. The second was that this election was not only about Brexit. While the absorption of Ukip and leave voters helped drive the Conservative vote share to its highest figure since 1983, the Labour vote grew among almost every other demographic.

But even if May’s gamble had worked domestically, it would have given her very little additional leverage with her negotiating partners in the EU. A huge majority would have given her more room for manoeuvre at home, but would have made little difference in Brussels.

But she did not. And she has now left herself no room to negotiate, and nothing to use in the negotiations. Big Ben isn’t the only clock ticking. May triumphantly triggered Article 50 before the election, and now the first round of talks with the EU over Britain’s exit will begin on June 19 – while May is still haggling with the DUP over the terms of propping up her minority government.

Contrary to the fantasies of the flag-wavers and chest-beaters, Brexit was always going to going to put the little in Little England rather than the great in Great Britain. But nobody realised our influence would shrink quite this fast and leave us looking quite this small.

If the old rules of politics have been laid to waste, we do not yet know what new rules will prevail. Much has changed since last week. Theresa May’s popularity ratings have plunged to -34, roughly the level of Corbyn’s last November, while now as many people have a favourable view of Corbyn as not. Survation, the pollster that came closest to predicting the election outcome, now puts Labour five points ahead of the Tories.

But much remains the same. The PLP has most of the same members; the same political journalists and columnists write for the same newspapers, and the same proprietors still own them; Corbyn still has the same flaws as a leader that he had before; and Theresa May, at least for now, is still the prime minister. Given the determination with which the mere possibility of this surprise result was denied, dismissed and derided, we should be under no illusion that the reality will be any more palatable to many.

Conventional wisdom has now developed a new convention of its own: first it states the uncertain with great certainty, only to be proven wrong by events, and then it embarks upon a period of narrowly tailored and very public retraction, which always falls considerably short of genuine introspection. After acknowledging an error of prediction, there are no efforts to address the underlying logic that produced that error. Their contrition only lasts until their next mistake.

An election provides but a snapshot in time, and these times are no less volatile when the results go your way than when they don’t. This election indicates that progressive politics are possible, and that the neoliberal programme is not quite as unassailable as many feared. This moment may have the promise of becoming something more. But at this point, it is no more than a promise.

But what it does do, perhaps, is give us a chance to figure out how we got here. In the decade since the financial crisis, there has been a series of political challenges to the notion that the poor must pay for the recklessness of the powerful, and to the belief that xenophobia must be accommodated rather than confronted – from Occupy Wall Street and UK Uncut to protests against the harsh treatment of refugees. But we have not seen those ideas challenged electorally until now. In 2010 and, to a lesser extent, in 2015, the narratives of austerity and nationalism were conceded by Labour in a failed attempt to neutralise them.

As long as Labour only spoke the language of greater fiscal responsibility and tougher controls on immigration, it could compete on the terms of the right, but it could never win. Yes, these are important issues, and yes, this strategy worked in the 1990s. But in times of economic crisis, people want more from a centre-left party than the promise of managing the crisis better – they want the crisis to end.

By the time of the EU referendum last year, enough people felt abandoned by the mainstream parties that some of them – enough to make the difference – used the ballot as an opportunity to express their dissatisfaction. This election was the first time since the crisis that a mainstream party had offered principled opposition to austerity and shifted the conversation from immigration to investment in public services. We were told that voters would not buy it. We were told it was not possible. But when the clock struck 10, the tectonic plates shifted. And for just a minute, until we found our footing, we felt a little giddy.

Main photograph: Ian Forsyth/Getty

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.