WASHINGTON (Reuters) - U.S. President Donald Trump said on Saturday that the top-ranking Democrat on the Senate Veteran's Affairs committee should resign after the President's pick to lead the Veterans Affairs department withdrew from consideration amid numerous allegations.

On Thursday, Trump's physician Ronny Jackson withdrew from consideration to head the department after allegations that he had been lax with prescription drugs and drank alcohol on the job.

Trump had already singled out Senator Jon Tester, the top-ranking Democrat on the Senate Veterans' Affairs committee, who is up for re-election in November in Montana.

"Admiral Jackson is the kind of man that those in Montana would most respect and admire, and now, for no reason whatsoever, his reputation has been shattered. Not fair, Tester!," Trump added.

In a statement, Tester said he would "never stop fighting."

“It's my duty to make sure Montana veterans get what they need and have earned, and I'll never stop fighting for them as their Senator," Tester said.

Jackson, a U.S. Navy rear admiral who has been physician to three presidents, has categorically denied the accusations as false. Later at a campaign rally in Washington, Michigan, Trump suggested he had damaging information on Tester.

"Tester started throwing out things that he's heard, well I know things about Tester that I could say too," said Trump. "And if I said them, he'd never be elected again."

Trump earlier tweeted that he had been told by the Secret Service that the Democrat's statements on Jackson were untrue.

"There were no such findings. A horrible thing that we in D.C. must live with, just like phony Russian Collusion. Tester should lose race in Montana. Very dishonest and sick!" he tweeted.

Democrats said more than 20 people, whose names they withheld, said Jackson had prescribed himself medications, got drunk at a Secret Service party, wrecked a government vehicle and once could not be reached on a work trip to provide medical treatment because he was passed out drunk in a hotel room.

Tester had said Jackson's nickname was "the candyman" because of his willingness to hand out prescription drugs.

The White House said on Friday it had looked through records of motor vehicle incidents involving government vehicles and found only three that involved Jackson. None of the records noted alcohol use. Reuters also reviewed two years of audits of the White House medical unit pharmacy, which did not show any major issues.

Separately, the U.S. Secret Service said it had no records to back allegations that its personnel intervened to stop Jackson from disturbing former President Barack Obama during a foreign trip in 2015.