FORT WORTH (CBS 11 NEWS) – Five days after the release of detailed sexual harassment claims against Tarrant County’s top prosecutor, Republicans and Democrats started calling for his resignation. District Attorney Joe Shannon quickly responded by saying he had no intention of stepping down.

County Tax Assessor Ron Wright said he notified Shannon Wednesday, shortly before releasing a strong letter saying the 72-year-old DA should leave office. Wright’s letter said the tawdry claims against Shannon cast a cloud over the office, the public’s confidence, and Shannon’s ability to effectively do his job. (Click here to download the entire letter).

It was not a decision Wright said, that he came to lightly, but he expected it might open the door for others to join him.

“The cloud’s not going to go away,” he told CBS11. “The controversy’s not going to go away. It’s not something I can fix, or the court can fix, the only one who can fix this is Joe Shannon.”

Shannon was accused in April of sexually harassing former co-worker Sabrina Sabin for up to four years. The county paid Sabin $375,000 in September to settle the claim out of court. The county paid attorneys to fight the release of details of the case, but the State Attorney General ruled earlier this month some documents, including Sabin’s hand-written notes of Shannon’s behavior, were public record.

Sabin’s accusations of lude comments and physical contact drew the most attention. However, two notations about race, and politics, are now also fueling calls for Shannon to give up his job.

Sabin noted “the blatant disregard for sex, race or other inappropriate jokes. The term democrat was used daily to refer to particular individuals in our cases. It was not being used as a term of endearment.”

Tarrant County Democratic Party Chairman Steve Maxwell said that was just one more reason for Shannon to step down.

“If you read through the rest of that diary it’s chock full of comment after comment that’s either racist, or sexist, any way you look at it,” Maxwell said. “That’s not what we need in an elected office.”

Maxwell went a step further, saying a grand jury investigation may be warranted, if Shannon’s comments indicated politics affected his choice of who to prosecute.

The L. Clifford Davis Legal Association, a group of primarily African-American attorneys in the county, stopped short of calling for Shannon’s job, but said the county commissioners should be asking more questions.

“If true, such jokes and comments by our top law enforcement prosecutor in the county is reprehensible, offensive and unbecoming of an individual who possesses the discretion to prosecute those individuals accused of crimes,” wrote President Nikki Chriesman.

Shannon declined to respond to the calls in person. He has not spoken publicly about the allegations since they were made. He has denied responsibility through statements from his office. Another statement Wednesday referenced the number of cases, life sentences and death sentences the office has handled.

Shannon did speak in person Wednesday to the chair of the county Republican Party Jennifer Hall. She asked for the meeting she said to find out more about the progress of the case, after taking a few calls from concerned elected officials and active party members. The party she said is taking the case seriously, but is not at this point calling for Shannon to quit.

Shannon’s term as DA runs through 2014. He has also been an adjunct professor at Texas Wesleyan University School of Law in Fort Worth. Sabin was a 2004 graduate of the school and listed Shannon as a reference on her county application. Shannon still has his adjunct position there but Texas Wesleyan had no other comment.

Jason came to North Texas after working as a reporter for four years in Orlando, FL. In 2010, he spent several weeks leading WFTV’s coverage of the Gulf oil spill and reported from Alaska on the lingering impact of the Exxon Valdez disaster as ...