For parens or braces highlighting, I suggest paren-mode or mic-paren-mode.Highlighting specific words is done with font-lock-add-keywords.The following snippet in your init file should color \quad in red in tex or latex buffers.

Still it messes up conditional Auctex highlighting when i add more complicated highlighting expressions. One of the examples would be when i change 0 to nil. I still don't know what it means , but that is an example.

The 0 means "highlight the whole match". A positive number means "highlight the nth group in the regexp". For example,the following code italicizes the word between braces in expressions of the form \begin{***}.

t means override any previous fontifications.See the documentation of the functions font-lock-add-keywords and font-lock-keywords for more details.There are some infos in the emacs manual about what is called "font lock". You can access the relevant part by evaluating (info "(emacs) Font Lock").