In view of gear transmission system’s complex nonlinear dynamic problem including dynamic transmission error, single-sided impact and double-sided impact, separation, rattling and chaos, taking into account nonlinear characteristics associated with gear clearance, bearing clearance, time-varying mesh stiffness and teeth surface friction, a new analysis method based on finite element method in solving multiple clearance coupling gear system’s nonlinear dynamic characteristics is first proposed. Take a single stage helical gear transmission system as the example, establishes its dynamic model by using ANSYS/LS-DYNA, has analyzed its nonlinear dynamic characteristics under supporting condition, and had studied the influence rules on system’s dynamic characteristics under the different speed and loading moment. The results show that finite element method is very efficient and reliable in solving complicated gear transmission system’s dynamic issue, and it will become a useful tool for further work on its nonlinear dynamic problem.