Yes, it is okay to say "That's dumb." Dumb means lacking intelligence or judgement, and as such can be used to describe people or actions. For instance: a teacher's decision to test students on material the teacher hasn't taught them yet is dumb. It's not the most articulate word, but it's an acceptable word. It's not the same thing as equating a group of people with an insult.

'Dumb' originally only referred to as another word for mute, or people who could not speak. So really it's a very similar situation, where a derogatory term for a minority has become the norm and gained a new definition.