The near three-hour grilling of Bob Diamond, the Barclays chief executive, turned out to be a frustrating time for George Osborne, as well as a setback for the reputation of the parliamentary select committee system. As for Bob Diamond, one suspects the Barclays board are not going to give him a massive severance payment on the basis of this performance. He never lost control during his cross-examination, but he never emerged as a man remotely in charge of the company he claimed to love.

Westminster operates to its own agenda and is less interested in the allegations of corruption at Barclays than whether Diamond would stand up the accusations made by Osborne – that the Brownites had put improper pressure on Barclays indirectly through the Bank of England to lower its Libor rate at the height of the banking crisis.

Some have claimed that such pressure would have been legitimate to stave off a greater evil – the entire collapse in banking.

Alistair Darling, the former chancellor and a good moral arbiter, had said earlier this week it would be reprehensible for any official or minister to apply such pressure. But he added it would be entirely desirable to apply a policy, such as a credit guarantee scheme, designed to ease conditions in the market and so lower the Libor rate.

But Diamond did not appear to stand up Tory claims that ministers had pressed Paul Tucker, the deputy governor of the Bank of England, to urge Barclays to post false rates at the height of the crisis.

Diamond's file note of his conversation with Tucker in October 2008 has been exhibit A in the Conservative case that the Brown circle had indeed put improper pressure on Barclays.

In that note, Tucker reiterated that he had received calls from a number of senior figures in Whitehall.

Indeed, as the inquiry started, Osborne ratcheted up the pressure by telling the Spectator magazine that the Brown circle were clearly involved.

Under cross-examination both by Michael Fallon, a Conservative party deputy chairman, and the committee chairman Andrew Tyrie, Diamond was woolly about whether these Whitehall figures referred to by Tucker were officials or ministers. He told Tyrie: "In my mind, people in Whitehall are officials." But later in an exchange with Fallon, he changed that to say: "What he [Tucker] was trying to tell me is: 'Bob, there are ministers in Whitehall who are hearing that Barclays is always high that could lead to the impression you are not funding yourself'."

Later he said: "I would only be speculating if I said who I thought they were. My recollection is Paul did not say to who he was referring."

The issue of the identity of shadowy Whitehall figures probably will only be cleared up by Tucker when he gives evidence to the committee.

But in a sense the identity does not matter because Diamond said he did not believe he was being asked to do anything wrong. He stubbornly insisted he did not believe the Tucker conversation represented an instruction to lower the rate. Nor did he accept it was a nod or a wink.

Even when Tyrie read out the passage of the Tucker memo – "Your Libor returns do not always need to be as high as they have recently" – Diamond refused to implicate either the Bank or Whitehall, or even accept he had been shocked.

Fallon pressed the point with Diamond, saying: "What you have written down here is that ministers or officials were in effect asking you to fiddle your submission."

Diamond responded: "I did not believe that, no."

Fallon pressed: "That is what it says here."

Diamond veered off, recalling that he responded to Tucker: 'Did you explain to minister the real story that other banks are posting rates below ours and yet are not borrowing money at those levels. It is not that our rates are wrong."

He said: "The most important thing to me in that note was the comment was that there was a perception in Whitehall that our rates were high," adding that the worry he shared with John Varley, the former chief executive, was that "if members of the government felt our rates were high relative to others and if they then took that to mean that we were having trouble funding".

His greater fear was that Whitehall might nationalise Barclays, just as he was raising £6.7bn in equity during the most momentous week in the history of Barclays.

He even pointed that although Barclays' Libor rates did indeed fall the day after the Tucker call, its Libor rate did not fall relative to the other 15 banks.

Nor could he explain why his juniors had come to believe an instruction had been issued by the bank.

So the Tucker memo has not yet become the smoking gun implicating the Brownites that the Conservatives had craved.

The Conservatives' current chief quarry is Shriti Vadera, Gordon Brown's close aide and a business minister at the time. Again Diamond did not help in his evidence. He simply said that "Shriti was very involved in the recapitalisation of the banks", but added he had not seen her that much.

She, like Ed Balls and the former city minister Lord Myners, categorically insists she had no conversations with Tucker or any other regulators to try to manipulate the rate down.

She said: "It was a completely legitimate concern of government and regulators to worry about the access to credit and the cost of credit to the real economy – that was what the financial crisis was about. In fact, it still remains something of an issue today, which is why even this government has put in policy interventions to ensure that small businesses can borrow at better rates from the market – that is the role of the government, and its job particularly at a time of crisis.

"That shouldn't be confused in any way with the actions of people who, as I understand for many years from this report, were manipulating how they set Libor."

The wider question is why Osborne is so determined – as he was in the Spectator – to put the Brownites in the frame. He told the Spectator: "As for the role of the Labour government and the people around Gordon Brown, well, I think there are questions to be asked of them."

It is possible that Osborne has access to some as-yet-unpublished information that is leading him to stoke all this so hard.

But he is also taking a high personal political risk. He is attacking the Brownites in the one area where they had a signal policy success – saving the banks from total collapse. There seems an absolute determination to prove that the banks ended in such a parlous state purely due to the lax regulatory regime of Ed Balls as city minister, rather their own moral and managerial failings.

But afterWednesday's evidence, it seems simpler and more accurate to blame the bankers rather than putting Brown, Balls, Vadera and Myners in the dock. That quartet may have misunderstood the markets, but they do not, on current evidence, look like they tried to rig them.