Boris Johnson, the London mayor and conservative candidate, has denied being "remotely sexist" following allegations that he is "disrespectful and patronising" to female politicians.

Johnson, who is seeking re-election on 3 May, rejected the claim after being confronted by a compilation of public exchanges made during his four-year term to highlight concerns over his "disrespectful, patronising way at meetings" towards female assembly members that "you do not display when dealing with male assembly members".

Jennette Arnold, Labour chair of the assembly, wrote to Johnson on 16 March after discussing the issue with all eight women cross-party members on the assembly. The Conservative mayor responded within three days to deny his behaviour was any different than towards the men on the 25-strong assembly and claimed some members were seeking to make a political point.

"I have not been more robust towards female rather than male assembly members and I do not believe I have been remotely sexist," wrote Johnson.

The Conservative candidate recently told the parenting website Mumsnet he was "a feminist", and the latest YouGov polling indicates that he is enjoying stronger support from female than male voters in the runup to the mayoral election on 3 May.

But female colleagues on the assembly appear less impressed with his performance.

Jenny Jones, the Green party mayoral candidate who has served on the London assembly since 2000, highlighted the correspondence as contenders prepare to debate the concerns of female voters at a hustings organised by the Fawcett Society on Monday.

Jones said: "He tends to treat men with a much more jokey matey attitude, whereas he tends to put us down and say things like 'I think you've added up your figures wrongly'."

She has accused Johnson of complacency over his decision not to attend the event on Monday night, entitled What About Women?

Johnson has cited an unfortunate diary clash, though his campaign team was unable to confirm exactly how he would be using his time instead.

He has asked Victoria Borwick, the only woman on the 11-strong Conservative assembly group, to represent him and take part in a debate alongside Labour mayoral candidate Ken Livingstone, Liberal Democrat candidate Brian Paddick, and Jones. But the move will prove embarrassing since an interchange between him and Borwick 18 months ago in the City Hall chamber is among the examples used by Arnold to underline her point.

Borwick urged him not to "trivialise" her question after he repeatedly interrupted her and tried to finish her question on rerouting buses – a plea that appeared to fall on deaf ears, according to the transcript.

Other exchanges drawn from the past four years include Johnson answering a question from Jones with "blah blah blah fishcakes" and on another occasion telling her to "ask better questions".

Johnson told Labour member and former deputy Labour mayor Nicky Gavron that she was talking "drivel" and "chuntering", and declared that Labour's Joanne McCartney should be a special constable because "she has the kind of air about her" .

It was a case of "rhubarb rhubarb rhubarb" when Labour member Val Shawcross challenged him on dropped manifesto promises.

Presented with the selection of quotes, Johnson hit back at the charge levelled by Arnold and insisted in the letter dated 19 March that the monthly mayor's question time held with the 25-strong London assembly each month was conducted "amid constant sledging and abuse" from opposition members, where "members of both sexes have been aggressive to me".

Caroline Pidgeon, Lib Dem deputy mayoral candidate, who is also standing for re-election on the London assembly, said: "I have not often experienced it myself but certainly I have observed the way he speaks to other female assembly members and it is patronising to say the least."

Lib Dem colleague Lady Doocey said: "My instinct is that he does it without meaning to do it, but it is insulting. That's the problem. I don't think he is sitting there thinking 'I'm going to be nasty to them because they are women' but there is no question in my mind that he treats women on the assembly in a very different way to men, and that is regrettable. I don't think he does it deliberately."

The Fawcett debate will take place on Monday following a report produced by the non-political society that draws on official figures to highlight the fact that women are less likely to work, earn less than men when they do, and are more likely to be sexually assaulted in the capital than anywhere else in the country.

Ceri Goddard, Fawcett Society chief executive said on the eve of the debate that she remained disappointed that Johnson had not taken the opportunity to "step up and show his commitment" alongside the other candidates to debate their policies to improve women's lives. "In that contex,t we have left the door open for him to join the hustings should he choose."

Pidgeon said it was a "bit rich" that Johnson asked Borwick to stand in for him following the decision of the Conservative associations to push Borwick down from second to third place on the party list for the elections behind two male colleagues in an election conducted under the additional member system.

While the latest YouGov polling shows Johnson is likely to be re-elected, it also suggests Tories may lose one of their 11 seats and end up with an all-male assembly group in the London assembly elections held the same day.

"[Borwick] has worked very hard, particularly on the London assembly and she has fully taken part in other committees," said Pidgeon.

Borwick and Arnold could not be reached for comment.

A Tory spokesman said: "The Conservative party is confident we can get the top three list candidates elected on 3 May with a strong Conservative vote."