The influence of rotational speed fluctuation and the phase shift between forward and backward contact on mesh point position are analyzed, combined with boundary conditions of single tooth meshing, double tooth meshing, profile modification and analytical method of calculating mesh stiffness, the nonlinear mesh stiffness model is established which can achieve real-time feedback with nonlinear model of spur gears and produce more precise mathematical model of gear meshing process. A two-degree-of-freedom nonlinear model of spur gear pair with input torque fluctuation is established by taking the backlash and nonlinear mesh stiffness into consideration. Based on the nonlinear dynamic model, the influence of profile modification on dynamic behavior of spur gear pair is studied. The resluts show that the amount of profile modification significantly affect the dynamic characteristics of spur gear pair, and existing an optimal amount of profile modification to make the dynamic load factor reach to a minimum. When the amount of profile modification exceeds a certain critical value, single-side-impact or double-side-impact occur and gear dynamic loads significantly increase. Increasing the length of profile modification can reduce the minimum value of dynamic load factor under certain input torque. The maximum amount of profile modification should be the combined tooth deflection evaluated at the highest point of single tooth contact under the minimum input torque.