This study focuses on the fault mechanism of the four-freedom gear system with tooth broken. Considering the unbalance caused by breakage and the effect of the defect on the mesh stiffness, the gear-rotor-bearing dynamic model of the gear system is established. The vibration features of simulated signals with toot broken failure are analyzed by time domain, phase contrail Poincaré mapping and frequency spectrum, which reveals the vibration response caused by tooth broken failure and the effect of tooth broken evolvement on the dynamic response . At last, mechanism research results are proved by the vibration characteristic of fault signals measured on gearbox experiment, which shows the dynamic model with tooth broken effective and the analysis results can supply a theory base to tooth broken fault diagnosis of gearbox.