We experimentally investigate the influence of vestigial sideband (VSB) filtering in a 40-Gbit/s optical system. Two kinds of optical filters and different pseudorandom bit sequences (PRBS) are used in experiment. The results show that optical filter ramp and PRBS length will cause different impacts in a VSB carrier suppressed return-to-zero system.