In order to reducing the vibration of herringbone gear transmission system more effectively, vibration reduction design methodsare put forward respectively from vibration excitation source of herringbone gear transmission system (tooth meshing quality) and vibration transmission path (box structure). Tooth surface three-dimensional modification is choosen to optimizing the vibration between the herringbone gear meshing teeth under multi-loads. Topology optimization is carried out target with static (box deformation) and dynamic (lower natural frequency) in order to improve the gearbox dynamics structure, and the optimization results can indicate which parts of redundant material and which structure needs to be strengthened. Accordingly, gearbox structural size is optimized to obtaining minimum acceleration of gearbox feet. Example calculations and experimental results show that the method of tooth three-dimensional modification reduce vibration of 18.9% by experiment test and 20.5% by theoretical analysis，the method of gearbox structural optimization reduce vibration of 12%. The better effect of 27.3% decline is obtained by combination of two methods.