You'll probably just be fine. Just calculate the power dissipated in your circuit. It's the square of the voltage across the pot divided by its resistance and make sure it's nowhere near 50mW. Personally I'd go with a 5x margin of safety for a small trimmer.