Fewer than half of Barack Obama’s biggest promises as president have been kept, according to a Guardian audit published on the eve of his final State of the Union address.

The analysis of six preceding state of the union speeches, two inaugurals and Obama’s first address to Congress in 2009 reveals a steady decline in both the number of policy pledges made by him, and their success rate, during an ambitious presidency marred by fierce battles with lawmakers.

The White House chief of staff, Denis McDonough, acknowledged this week that Tuesday’s address will be unusual in avoiding many of these concrete proposals for the final year in office and would focus instead on what needs to change in the political climate before his vision for the country can be realised by a successor.

“He gathered us together late last year and he said, look, he doesn’t want this to be your traditional policy speech that outlined a series of proposals,” McDonough told ABC on Sunday. “Rather, he wanted to step back and take a look at the future of this country, the challenges we’re going through.”

While advisers are still hopeful of achieving some reform in 2016 through executive action, a review of 27 key policy areas over the course of Obama’s nine previous speeches to Congress reveals just how much his election promise of “hope and change” has been stymied by deadlock with Republicans.

Progress on immigration, gun control, tax, and overseas troop withdrawals has been slowest, while Obama’s successes in tackling the economy, healthcare reform, climate change and diplomacy have proved far wider than many critics expected.

A mixed record on Wall Street, civil liberties, and counter-terrorism arguably reflects the difficult political compromises faced at home and abroad.

While any assessment of the overall success of Obama’s two terms in office requires subjective judgments, the annual State of the Union provides a unique window into the policy priorities that the president and his speechwriters have chosen to publicise in any given year.

And as the president prepares tonight for arguably his final big set-piece speech setting America’s direction of travel, it is a moment to reflect on his legacy.

Many themes return frequently in the 132 separate pledges identified and assessed by the Guardian spanning his two terms. Tax reforms designed to encourage American multinationals to invest at home and simplify the tax code have been a feature of every Obama State of the Union, for example. Others have been presented with steadily less conviction over time: appearing first as bold predictions, and eventually only as requests or suggestions to Congress.

But the analysis attempts to assess the ultimate success of each firm promise in the year it is made counting setbacks and achievements. A score between zero and five has been awarded, with zero for virtually no progress, five for almost complete success and a sliding scale in between.

Such analysis reveals a marked difference between the soaring rhetoric at the start of each term in the office, and the more successful and pragmatic speeches he made when faced with the more focused economic challenge of dealing with the aftermath of the financial crisis.

The most ambitious speech was the 2009 inauguration address, delivered on the steps of Capitol Hill before a record crowd of some 1.8 million, which contained at least 20 major pledges on everything from education to the economy.

It was also Obama’s most optimistic: a time for him to “proclaim an end to petty grievances, recriminations and dogma that strangle our politics” and fulfil the promise of bipartisan compromise on which he campaigned.

“The challenges we face are real,” acknowledged Obama. “They are serious and they are many. They will not be met easily or in a short span of time. But know this America: they will be met.”

The language was at its most poetic then too, with Obama signalling his promise to reduce inequality, for example, more elliptically than in later speeches: “The nation cannot prosper long when it favours only the prosperous”.

But subsequent events suggest much of this optimism that Washington could be united behind his progressive agenda was misplaced, and Guardian assessment of the pledges suggests an average score of 2.4 out of 5, a success rate of barely half.

More progress was made when the president delivered his second big speech to the nation a month later, in an occasion many saw as his first State of the Union, even though technically it was just an address to Congress.

In this, and the 2010 state of the union to follow, Obama focused heavily on fixing the economy: promising job growth and recovery in the housing and stock markets. Critically, Obama also used the Democratic majority in both the House and Senate to drive through his healthcare reform – perhaps his most lasting achievement, that leaves 2010 as a high watermark for deliverable public pledges.

By contrast, the challenges of governing after Republicans took back control of the House in 2010 were beginning to catch up with the president by the 2012 election year state of the union – noticeable for 13 big pledges, and ultimately an average success rate of just 2.2 on the 0 to 5 scale.

More significantly, foreign policy setbacks have weighed heavily on the assessment of Obama’s achievements, particularly his promise to end the wars in Afghanistan and Iraq and bring all US troops home. Though temporarily successful, both policies have had to be reversed, as the rise of Isis and resilience of the Taliban show the president’s broader counter-terrorism achievements to be worryingly fragile.

Obama has frequently acknowledged that his ambitious agenda at home and abroad was always likely to see setbacks, a fact borne out by the average success rate across all nine speeches of 2.2 out of 5.

“I campaigned on the promise of change – change we can believe in, the slogan went,” he recalled in 2010. “But remember this – I never suggested that change would be easy, or that I could do it alone.”

But few of his supporters would have imagined that a candidate who fought so hard against the legacy of George W Bush would be sending US troops back to Afghanistan and Iraq in 2016, or still struggling to close the Guantánamo Bay detention centre.

On the other hand, historians may view the turnaround in the US economy – including the longest period of sustained job growth on record – and the dramatic reductions in the uninsured under Obamacare as more than compensating for uncontrollable events abroad and the stubbornness of opposition in Congress.