Based on the method of Prohl-Myklested, a formula that was used to calculate equivalent torsional stiffness of massless elastic shaft was proposed. Because sections of disks of automotive transmission helical gear system had little change compared with that of shaft, it was simplified lumped parameter model which had fewer degrees of freedom by that method. And model analysis of simplified model was carried out. Stiffness matrix of shaft-disk system was proposed in the case of different bending stiffness of shafts in both ends of disk. Moreover, equations of motion of helical gear rotor system with lumped mass at ends, asymmetric elastic support and different bending stiffness and torsional stiffness of each shaft were established. Then numerical integration method was employed to study the non-linear characteristics of system. The results show that: the model of automotive transmission helical gear system reduced its dimensionality by the method of Prohl-Myklested improved can accurately reflect the dynamics responses of the system. Coupling backlash with time-varying stiffness causes the system to appear complex non-linear phenomena. Meanwhile, characteristics of backlash fault are extracted in order to provide basis for the gear fault diagnosis.