Program refactorings in IDEs are commonly implemented in an ad hoc way, since checking correctness with respect to a formal semantics is prohibitive. These tools may perform erroneous transformations, not preserving behavior. In order to detect these errors, developers rely on compilation and tests to attest that behavior is preserved. Compilation errors, for instance, are simple to identify by tools. However, changes in behavior very often go undetected. We propose a technique for generating a test suite that is specific to pinpoint incorrect refactorings. In each refactoring, we identify program parts that are common to the program before and after refactoring, and automatically generate a comprehensive set of unit tests for the initial program aided by a test generator. As such, the create test suite is run on the program before and after automatic refactoring, which does not require also to refactor the test suite itself in our technique, as may occur sometimes in traditional refactoring scenarios. Our technique is evaluated against a benchmark of 16 refactoring cases which present errors when performed by mainstream refactoring tools. This evaluation has been successful in detecting more than 93% of those errors.