House oversight committee Chairman Darrell E. Issa said Thursday he’s baffled that the Justice Department declined to prosecute a government employee who apparently knowingly pried into tax records of a political candidate or donor, and said there should be a way for victims to know their rights have been violated.

He also said letting state officials have access to federal tax records can be dangerous, and that’s one reason to try to rein in the health care law, which will expand access to states that plan on running their own insurance exchanges.

Mr. Issa made the comments after The Washington Times reported this week that the Treasury inspector general for tax administration had found at least four cases in which a candidate’s or donor’s tax information was accessed improperly.

In one case, the investigator said the violation was willful and referred it to the Justice Department, which declined to pursue the case — a move Mr. Issa said was worrisome.

“We are not just disappointed, but dismayed,” Mr. Issa, California Republican, told The Times, speaking just after he finished a hearing with Inspector General J. Russell George looking at the intrusive IRS scrutiny of conservative groups applying for tax-exempt status.

The Times reported on former Senate candidate Christine O'Donnell, who received a voicemail message this year from a Treasury Department official alerting her that her “personal federal tax info may have been compromised and may have been misused by an individual.”

She said Dennis Martel, the official, told her an official in Delaware’s state government improperly accessed her records in 2010 on the same day she announced her plans to run for that state’s U.S. Senate seat.

Investigators for Sen. Chuck Grassley of Iowa, the ranking Republican on the Senate Judiciary Committee, said the problem was likely a backdoor system that allows state officials to look at federal tax records as part of their own investigations.

Ms. O'Donnell has tried to find out what happened to her information, but has been stymied. Investigators told her the probe has been closed, but cannot say more.

At Thursday’s hearing, Mr. George said federal law prohibits him from giving victims details about their cases.

“Justice will almost always say that they go after cases where they get a conviction, but as you know, they certainly go after a great many cases in which conviction is unlikely,” he said, pointing to the case the department brought against former Major League Baseball pitcher Roger Clemens.