Gear pitting fault causes change in non-Gaussian components of vibration signals. This change can be quantitatively described by higher order spectral analysis. Measured vibration signals of the gearbox, and made study on the development process of gear pitting fault in depth. At first retained the sideband components and the random components of vibration signals by using Gabor filter. Secondly analyzed the bispectral results and researched the change of nonlinear non-Gaussian characteristics of vibration signals, then extracted the non-Gaussian intensity characteristic values. At last made the fault trend analysis and set fault threshold value by ‘3σ criterion’. The results show that the non-Gaussian intensity characteristic value is sensitive to gear pitting fault, and it can reveal the trend of fault development. It’s conducive to the failure alarm and life prediction of gears, and has practical significance to condition monitoring and fault diagnosis for gear transmission system.