The parenthesis are passed to the test command as arguments. The [ ... ] expression is just another way to type the test command. The only difference is that the last argument for the [ command must be ].

So you should escape the parenthesis just as any other arguments you pass to a command to avoid the shell interpretation:

if [ \( $# -lt 2 \) -o \( $# -gt 3 \) ]

Alternatively, use single quotes:

if [ '(' $# -lt 2 ')' -o '(' $# -gt 3 ')' ]

From the info page:

`test'
`['
test EXPR
Evaluate a conditional express ion EXPR and return a status of 0
(true) or 1 (false). Each operator and operand must be a separate
argument.