In the end, the night was not to be his. Beto O’Rourke, an obscure Democratic member of Congress from the border town of El Paso who has risen to be one of his party’s new superstars, conceded defeat in his insurgent bid to unseat Ted Cruz from a US Senate seat in Texas – but in words that sounded like those of the victor.

“I believe in you, I believe in Texas and I believe in this country,” he told a crowdin his home town of El Paso. “I’m as inspired, I’m as hopeful as I’ve ever been in my life. Tonight’s loss does nothing to diminish the way that I feel about Texas or this country.”

Thousands of O’Rourke supporters had waited hours for their champion to appear, filling a baseball stadium. Though the mood grew sombre when news came in of his defeat, it lit up when he took to the stage shortly after 10pm with chants of “Beto! Beto!” rising above the crowd.

Such was the distance he moved the political dial in Texas – his was the best showing by a Democrat in a US Senate race for 20 years – he is certain to have a prominent voice in the party from now on. He took a state that had been written off by progressives for decades, and virtually single-handedly wrestled it into play.

The trailblazing candidates who broke barriers in the 2018 midterm elections – video

“O’Rourke raised more money than any other Texas Democrat, electrified crowds like no Texas Democrat (or Republican) in recent history, and breathed new life into a moribund Democratic party. While Beto-mania has bitten the dust tonight, he still has a bright future ahead of him,” said Mark Jones, a political scientist at Rice University.

Julian Castro, Obama’s housing secretary and the former mayor of San Antonio, told the Guardian that by coming so close, O’Rourke had given a major boost to Democratic hopes in Texas. “His achievement is to show that Texas, with its 38 electoral college votes in presidential elections, is back in play.”

O’Rourke has electrified crowds and breathed new life into a moribund Democratic party

Castro added: “This is a wake-up call for this state. Democrats can go much further and faster than anyone had thought.”

Ted Cruz celebrated hanging on to his Senate seat in front of several hundred supporters had gathered in a hotel ballroom in one of Houston’s swankiest districts

When Cruz entered the ballroom he high-fived his fans as they chanted “U-S-A!” and embraced his wife, Heidi, before taking the stage. “Texans came together behind a commonsense agenda of low taxes, low regulations and lots and lots of jobs. Securing the border and keeping our communities safe and defending the constitution and the Bill of Rights,” he said.

The 47-year-old also called for more civility, respect and dignity in politics, which sounded a touch hollow from a man who only two weeks earlier had appeared on stage at a rally held by Donald Trump in Cruz’s honour in which the president deployed his usual repertoire of ultra-partisan insults and false claims.

Cruz – Trump’s sworn enemy when they fought for the 2016 Republican presidential nomination – genuflected before him and ran a campaign based almost entirely on turning out a base of white, conservative Trump loyalists.

As the battle heated up, Cruz, relying on suburban and rural voters, relentlessly attacked his Democratic opponent as too liberal for Texas, portraying him as far-left Hollywood celebrity bait, a Bernie Sanders-style socialist out of touch with traditional values – that is to say, a full-throated embrace of God, guns, fossil fuels, low taxes and limited government.

Cruz hammered O’Rourke on his pro-immigration policies, warning of the dire consequences of allowing “caravans” of Central American asylum seekers into the country in an echo of Trump. He also turned into an attack ad a viral video in which the Democrat said there was “nothing more American” than kneeling during the national anthem as NFL players had done in protest at police brutality.

Nonetheless, when the result was certain and O’Rourke posed no more threat, Cruz took time to congratulate his opponent. O’Rourke, he said, had “poured his heart into this campaign, he worked tirelessly, he’s a dad and he took time away from his kids. Let me say to all of those who worked on his campaign, those who were inspired, that I am your senator as well and my responsibility is to represent every Texan.”

Before Cruz gave his victory speech in Houston, his father Raphael appeared on stage. “The message is loud and clear,” he told the Republican crowd. “Texas remains solid red!”

But that message is not at all clear after a bruising campaign that saw the Democratic underdog come far too close to winning the Senate seat for comfort. The last time a Democrat successfully did that was in 1994.

O’Rourke began his David versus Goliath mission to topple Cruz less than two years ago with a staff of just two – both old friends from El Paso – travelling in a rented sedan. He started from scratch in a state with next to no Democratic party infrastructure, criss-crossing the state to stump in all of its 254 counties – no small task in Texas, which is bigger than France.

Wherever he went, he planted seeds of a new Democratic infrastructure – something that has been sorely lacking in Texas since the 1990s. They began recruiting volunteers, often young and inexperienced but energetic and eager, who grew into an army by election day, numbering 25,000.

O’Rourke’s team created 727 “pop-up” offices, converting volunteers’ homes into hubs of activity. By the end they had knocked on almost 2m doors.

The Democratic candidate went to extreme lengths to mobilise every potential vote. When the Guardian reported on an Hispanic young man in Gonzalez that had never voted and had no intention of starting now, he dispatched his field officer from 70 miles away to register the individual and encourage him to cast his ballot.

Financially, he also tore up the traditional rulebook. He refused from the beginning to accept money from big donors or political action committees, preferring instead to rely on the beneficence of his passionate supporters.

The gamble worked – O’Rourke smashed previous US Senate fundraising records, hugely outgunning Cruz by drawing in about $70m from more than 1m small online donations to his opponent’s $30m.

In the end, the vastly superior ground game that the Republican party has built up over many years was simply too much for one individual politician to overcome.

But the narrow result still represents a stark improvement for the Democrats in Texas, a state that turned majority minority in 2004 and with its rapidly growing population holds enormous weight in US electoral politics. Were the Democrats able to win Texas in presidential races they would effectively banish the Republicans from the White House.

Perhaps O’Rourke’s most important achievement has been to prove that groups that have been written off as potential voters in Texas – notably young people aged 19 to 29 – can be brought to electoral life. His campaign led to a surge of voter registration that added 1.6 million voters to Texas voter rolls.

Having registered them, O’Rourke’s army of volunteers then persuaded them to actually go to the polling stations, leading to a massive surge of early voting that saw almost 5 million Texans vote early – more than the total who cast ballots in the last midterm elections in 2014.

Cynthia Valdez, a customer service assistant in El Paso, is one of the new legions of Democratic voters in Texas unleashed by O’Rourke. She is 27, but voted for the first time in early voting last month.

She said she was partly motivated by Trump. “He is not for the people, he is only interested in himself, his friends and the rich. He only cares about money.”

And it was partly motivated by O’Rourke. “He’s for the average person. He cares about us.”