Established 6 DOF nonlinear dynamic model of spur gear with consideration of friction. Using A-type gear in FZG test rig as example, calculated realistic time-varying mesh stiffness, analyzed time-varying friction coefficient based on the mixed lubrication model. For three gear system with various tooth surface roughness, the dynamic response of the gear meshing such as normal load, friction force, bearing force and the dynamic transmission error (DTE) was comparatively investigated in time and frequency domain by numerical simulation method, and the vibration test experiment on different gear pair with perfect and damaged tooth surface was conducted to validate the analyzed result finally. The result shows that: tooth friction will inhibit the vibration in line of action (LOA), increasing tooth surface roughness will reduce the DTE, but exacerbate vibration in off-line-of-action (OLOA), hence to increase system vibration amplitude, results provide a practical reference for diagnosis on tooth surface micro-damage tooth surface.