The recent decision to ban a Ten Commandments display at a Oklahoma courthouse has sparked concern among religious rights groups.

A three-judge panel of the 10th U.S. Circuit Court of Appeals ruled Monday that the monument on the lawn of the Haskell County Courthouse endorses religion and must be removed. The ruling overturned a previous judge's decision to keep the Ten Commandment display in place.

"Whoever was the judge in this, I feel sorry for him on Judgment Day," Haskell County Commissioner Mitch Worsham said in response.

Now, Alliance Defense Fund attorneys are considering a possible appeal of the recent decision, which could lead to a hearing before the Supreme Court. The Christian legal group argued against the case and says people shouldn't have to give up their religious heritage "simply to appease someone's political agenda."

"There is no difference between this Ten Commandments display and the one at the Texas state capitol that was upheld by the U.S. Supreme Court nearly five years ago," said ADF Senior Counsel Kevin Theriot. "This one, like that one, was donated and paid for by a private individual and displayed among numerous other monuments to veterans, pioneers, and others."

"The emotional response of a single, offended passerby does not amount to a violation of the Establishment Clause," he added.

The lawsuit was filed by the American Civil Liberties Union on behalf of a man who said he was offended by the display.

ADF can ask the panel to review their decision or appeal directly to the U.S. Supreme Court.

Meanwhile, Commissioner Worsham says his county will now remove the display.