The reduction gearbox of medium speed mill is used as physical model. Mathematical model is established with the minimum volume as the objective function， the number of teeth， the gear modulus and the tooth width as the design variables， the strength of the gear and so on as the constraints. The optimized results are compared with the original data and the results show that the volume is reduced by 29%，which can economize material and realize compact configuration.