If the VU meters are classic, passive analog meters they’ll add a little distortion to any circuit they’re paralleled with. This can be worse in transformer balanced matched impedance setups, but it exists in modern opamp ones as well. Buffered electronic meters shouldn’t cause this distortion.

Whether it’s a problem is up to you. A lot of classic recordings were made with this kind of setup.