The Supreme Court said Monday it won't hear an appeal from the family on TV's "Sister Wives" challenging Utah's law banning polygamy.

The justices left in place a lower court ruling that said Kody Brown and his four wives can't sue over the law because they weren't charged under it.

A federal judge sided with the Browns and overturned key parts of the state's bigamy law in 2013, but an appeals court overturned that decision last year.

The Browns claim the law infringes on their right to freedom of speech and religion. The family said they should be able to challenge the law because the threat of prosecution forced them to flee to Nevada and still looms over them when they return to Utah.

Police investigated the family after their show premiered in 2010, but closed the case without filing any charges. The family argued in legal briefs that the state should not be able to thwart a constitutional challenge to the law "by changing its enforcement policy during the pendency of litigation."

Utah's law forbids married people from living with a second purported "spouse," making it stricter than anti-bigamy laws in other states and creating a threat of arrest for plural families. But state officials have followed a long tradition of not prosecuting polygamists unless they commit some other crime, such as child or spousal abuse, domestic violence or fraud.

The state says only 10 people were charged with violating the law between 2001 and 2011. Utah officials argue that the ban is important to protect vulnerable people from exploitation and abuse.

Kody Brown is legally married to Meri Brown, but says he is "spiritually married" to three other women. They live together in a plural relationship and belong to a religious group that believes in polygamy as a core religious practice.

About 30,000 polygamists live in Utah, according to court documents. The mainstream Church of Jesus Christ of Latter-day Saints abandoned the practice in 1890 and strictly prohibits it today.

An attorney for a TV's "Sister Wives" family says the U.S. Supreme Court decision not to hear an appeal of Utah's law banning polygamy won't end the larger fight by plural and unconventional families for equal status.

Lawyer Jonathan Turley said Monday in a statement posted on his blog that he and the Brown family are disappointed but not surprised by the decision that was issued by the high court without comment.

Turley emphasized that an appeals court ruling that stands wasn't made based on the merits of the Browns' assertion that Utah's law violates their rights of speech and religion. That court found Kody Brown and his four wives can't sue over the law because they weren't charged under it.

The family says it wanted to challenge the law because the threat of prosecution still looms over them.