WASHINGTON (CNN) -- Washington lawyer Jim Robinson is a former assistant attorney general and once served as a U.S. attorney in Michigan.

Jim Robinson, a D.C. attorney, is on the U.S. watch list by mistake and wants his name removed.

He's an American citizen who holds a high-level security clearance from the U.S. government. He's a onetime law-school dean, a husband and a grandfather.

And he's on the U.S. government's anti-terrorism watch list, which gets him delayed or stopped every time he tries to board a commercial airliner.

"I don't feel safer because I have to go through this hassle; I can tell you that," Robinson said.

The list has swollen to an estimated 1 million names, the American Civil Liberties Union said this week. The scale renders it "ineffective" and "likely useless" as a tool against terrorism, said Caroline Fredrickson, the ACLU's Washington legislative director.

"The list is unfair to travelers, unfair to law-abiding Americans and unfair to the security screeners who have to work in this ridiculous system," Fredrickson said Monday.

The federal officials who manage the list say the ACLU's claim is overblown and that only 400,000 names are on the list. About 5 percent of those are Americans, and most are not even in the United States, according to the Terrorist Screening Center, a division of the FBI.

Don't Miss

"We strive to have the watch list contain all appropriately suspected terrorists who represent a threat to the U.S., but only appropriately suspected terrorists," the center said in a written statement. "Independent government audits have recognized our ongoing efforts to constantly check watch list data to improve quality, reduce the number of misidentifications or mitigate their effects, and enhance traveler redress efforts."

Being on the list means he can't use some of the conveniences of air travel that many passengers now take for granted, such as electronic check-in, automated ticketing kiosks and curbside baggage checks. Instead, once he checks in, he has to wait in a line to watch airline agents get on the phone to verify "that I am not the James Kenneth Robinson who is the cause of my being on the watch list."

"On May 2, 2005, I filled out all their forms, made a copy of my passport, my driver's license, my voters registration card, put it in a package and sent it off to [the federal Transportation Security Administration] and never heard back," he said. "It certainly doesn't seem to have done me any good at all."

FBI spokesman Chad Kolton said inconveniences like Robinson's were unfortunate but "side effect of our efforts to fight terrorism." The majority of those people on the list have multiple names, he said.

The ACLU drew its 1 million figure from an October report by congressional auditors, who put the number of names on the list at more than 750,000, with about 20,000 names being added each month.

"This list is not only a bureaucratic nightmare that has captured millions of people in its web, but it diverts our security resources, so it actually makes us less safe," said Barry Steinhardt, director of the ACLU's technology program.

Among those added to the list recently is CNN Investigative Correspondent Drew Griffin, who learned of his new status in May. Griffin has done critical reporting on the agency, but TSA spokesman Chris White says that any connection between the two events "is absolutely fabricated."

Steinhardt says the list is so secretive yet so shoddily put together, it's hard to tell if how it is being used -- or abused.

"The truth is, we don't know how much is bureaucratic ineptness or how much is political retaliation," he said.