As an alternative explanation of the dimming of distant supernovae it has recently been advocated that we live in a special place in the Universe near the centre of a large void described by a Lemaitre-Tolman-Bondi (LTB) metric. The Universe is no longer homogeneous and isotropic and the apparent late time acceleration is actually a consequence of spatial gradients in the metric. If we did not live close to the centre of the void, we would have observed a Cosmic Microwave Background (CMB) dipole much larger than that allowed by observations. Hence, until now it has been argued, for the model to be consistent with observations, that by coincidence we happen to live very close to the centre of the void or we are moving towards it. However, even if we are at the centre of the void, we can observe distant galaxy clusters, which are off-centre. In their frame of reference there should be a large CMB dipole, which manifests itself observationally for us as a kinematic Sunyaev-Zeldovich (kSZ) effect. kSZ observations give far stronger constraints on the LTB model compared to other observational probes such as Type Ia Supernovae, the CMB, and baryon acoustic oscillations. We show that current observations of only 9 clusters with large error bars already rule out LTB models with void sizes greater than approximately 1.5 Gpc and a significant underdensity, and that near future kSZ surveys like the Atacama Cosmology Telescope, South Pole Telescope, APEX telescope, or the Planck satellite will be able to strongly rule out or confirm LTB models with giga parsec sized voids. On the other hand, if the LTB model is confirmed by observations, a kSZ survey gives a unique possibility of directly reconstructing the expansion rate and underdensity profile of the void.