Senior Government figures have challenged Coalition frontbencher Christopher Pyne to reveal the extent of his contact with the political aide who has made sexual harassment and fraud allegations against the Speaker Peter Slipper.

Fairfax newspapers report Mr Pyne, the manager of Opposition business, and James Ashby, the staffer who has filed court documents alleging Mr Slipper sexually harassed him and misused taxi vouchers, had drinks in the Speaker's parliamentary office a month before he made the allegations.

The papers claim Mr Pyne drank with Mr Ashby from about 9:30pm until after 11:00pm on March 19 and quotes Mr Pyne as saying he was merely "passing the time of day".

Treasurer Wayne Swan and Trade Minister Craig Emerson have seized on the claims as a double standard in the Opposition's attack on Mr Slipper.

"Mr Pyne has many questions to answer," the Treasurer said.

Mr Swan challenged Mr Pyne to release phone records proving he had not been contacting Mr Ashby to assist with the civil sexual harassment case against the Speaker.

"Did he phone Mr Ashby after he requested his telephone number? Is he prepared to provide his phone records and make those public if he hasn't phoned Mr Ashby?" Mr Swan said.

"The same standard that is being applied to Mr Slipper ought to be applied to the Opposition."

No Australian would believe that sitting with someone for almost two hours and drinking constitutes passing the time of day.

Federal Trade Minister Craig Emerson

Mr Pyne says he has had no telephone contact with Mr Ashby "at any time".

But Dr Emerson continued the attack on the Opposition frontbencher, accusing him of being coy about the specifics of his contact with the staffer.

"When asked about any knowledge that they had had in relation to the document prepared by Mr Ashby, both Mr Abbott and Mr Pyne have used a pre-agreed formulation saying they have no specific knowledge," he told PM.

"I have suggested that this is a weasel word. The insertion of the word 'specific' covers potentially a lot of facts that they may not want to be in the public arena.

"Why didn't [Mr Pyne] say at the outset, 'Yes of course I know Mr Ashby, I've spent up to two hours having a drink with him but I had no prior knowledge at all', rather than, 'no specific knowledge'?

"No Australian would believe that sitting with someone for almost two hours and drinking constitutes passing the time of day.

"Then there's the question of whether Mr Pyne subsequently rang and asked for Mr Ashby's mobile number. I understand that Mr Pyne is now saying he did not, but only a little while ago he was saying he could not remember."

'Political conspiracy'

In response to the allegations, Mr Pyne confirmed he met Mr Ashby three times in the months before Mr Ashby made his allegations against the Speaker.

But Mr Pyne says the Government is trying to create a fanciful "political conspiracy" about his contact with Mr Ashby and says that in his role as manager of Opposition business he is in "regular contact" with the Speaker's office.

"In the course of my work as manager of Opposition business, I have had cause to meet Mr James Ashby three times - twice in the Speaker's office and once when he came to my office to collect wine being given to a former Coalition staffer as a farewell gift," Mr Pyne said in a statement.

"On no occasion did he raise the matters canvassed in the Federal Court action that have subsequently come to light with me."

Mr Slipper has stood aside as Speaker until all the claims against him are resolved.

On ABC1's 7.30 last week, Mr Abbott said he was not aware of Mr Ashby receiving any assistance from the Coalition in preparing his court claim.