"The admission and apology by the IRS that the criteria used [in tea party applications] was not correct and inappropriate" is grounds for a suit, Sekulow said in an interview with ABC News because groups he represents are "still getting letters requesting information," and he believes, "if we don't file suit we won't bring an end to this."

ACLJ represents 27 tea party organizations that feel they were unfairly targeted by the IRS. Ten of the groups ACLJ represents still have pending applications to get tax-exempt status, while 15 did receive tax-exempt status.

ACLJ is preparing a lawsuit against the Department of Treasury and the IRS on behalf of at least 17 of those groups, Sekulow said -- and more groups, possibly even potential new clients beyond the 27 the ACLJ currently represents, could be added.

Sekulow said he plans on filing the suit this week.

"We are going to focus on a lack of standards and the treatment they were given and the wrong criteria utilized, much of what was in the IG report," Sekulow said.

Other government officials might be added to the suit, he said, including IRS agents named in the documents and department heads.

Who's Considering Suing?

One of the groups included under the ACLJ umbrella is the Linchpins of Liberty, which the president Kevin Kookogey told ABC News was set up as an educational initiative to provide high school and college students a grounding in conservative political philosophy. He said the group was singled out by the IRS for extra scrutiny, will "absolutely" be part of the ACLJ's suit and "fully intend to file" this week. The group's application for tax-exempt status is still pending and it has been "27 months and counting," according to Kookogey.

Larry Nordvig, the executive director of the Richmond Tea Party, wasn't as sure as Kookogey, but said the group was considering joining the suit and working with the ACLJ, as well. There is "more of a chance we will engage in legal action" after a congressional hearing on the matter Friday, he said, because he didn't feel like he got any answers. The Richmond Tea Party was approved for tax-exempt status in July 2012 after first filing for it in 2009.

"This is an application, this isn't an audit," Zawistowski said, referring to the long and detailed questionnaires his group was asked to fill out. "You were auditing us before we did anything."

The group's application for tax-exempt status was approved in December 2012 after first applying in June 2010.

Eric Wilson of the Kentucky 9/12 Project said the group is "considering" a lawsuit, working with the ACLJ and will be "part of the batch" when ACLJ files. It got IRS approval as a tax-exempt organization in March 2013 after first applying in December 2010.

Toby Marie Walker, president of the Waco Tea Party, also is working with the ACLJ. Although she is not completely certain, she expects her organization will be "most likely part of the ACLJ" group that files suit. Her group was approved for tax-exempt status in March 2013 after initially filing in July 2010.

Carole Waddell, the treasurer of the Waco Tea Party, said "we felt it was intimidation and harassment just because it tied up a lot of our time and kept us from doing other activities that we would have normally been doing in an election year last year.

Adrienne King, who heads up the Honolulu Tea Party, said it has agreed to be a plaintiff in the ACLJ case. Its tax-exempt status was approved in March 2012, after it first applied in December 2011.

Margie Drescher, the executive director of the OKC Tea Party in Oklahoma City, said it was part of the group that will file with the ACLJ. Its tax-exempt status was approved in July 2012. The group first applied in October 2010.

Not all of the groups planning on turning to litigation will be part of the ACLJ lawsuit.

Julia Hodges, who heads up the Mississippi Tea Party is "talking about" filing suit, but said the group was "not close to deciding."

Mary Beth Hutchins with Cause of Action is in the same boat, telling ABC News it was "not currently filing any lawsuits, but [its leaders] are considering the possibility of filing in the future."

The Tea Party Patriots, a national tea party group that represents smaller tea party groups all over the country, wrote a letter to the IRS Thursday saying it was "presently preparing lawsuits to be filed against the IRS on behalf of those organizations." It wrote it was advising the IRS of the development in order to make sure relevant documents were not destroyed, saying it wanted "to place a litigation hold on all documents, correspondence and other materials related to the 'targeting,' including all electronic information, emails (whether sent or received on government or personal email accounts or devices), correspondence, etc., all phone records, notes, and papers of the IRS employees who have been involved, at all levels within the Service."

Logan Churchwell, spokesperson for True the Vote, a group that says it was also targeted by the IRS, said the group will file a federal lawsuit on Tuesday, although that date may change.

Catherine Englebrecht, the president of True the Vote, said in an interview that she was told during the IRS application process by an analyst in Cincinnati: "I'm just following directions and the directions are coming from Washington."

Englebrecht said her group's application for tax-exempt status has been pending for three years, but during that time she was personally audited and even visited by agents from the Bureau of the Alcohol, Tobacco and Firearms.

"Initially, we were filing [a lawsuit] because we had not yet received any kind of determination," Englebrecht said. "Now, with the IRS' admission that certain groups were singled out, we want to fully explore every opportunity to make sure that what we suffered through the last three years is duly noted. ... I think it begins to point to a pattern of behavior that is very troubling."

True the Vote added that it was concerned that other Tea Party groups around the country were asked about their relationship with them in questionnaires.

True the Vote came under serious scrutiny during the election for its work monitoring polling places, with critics saying it was trying to suppress Democratic and minority voters.

Cleta Mitchell, an attorney with Foley & Lardner, LLP which is working with many of the groups that claimed unfair scrutiny, including True the Vote, said there are "hoards of lawyers reviewing the case law and statutes as we speak" -- but, "trying to find out the basis for damages" isn't simple and there isn't "always a remedy in the courthouse."