Through analyzing gearbox vibration signals under varying speed conditions of rotating machinery, a method of analyzing gear fault based on narrowband demodulation with frequency shift and spectrum edit was proposed here. Firstly, time-domain signal was sampled under varying speed condition of gearbox and was resampled at constant angle increments according to order tracking, converting into stationary signal in the angle-domain, which can overcome the frequency blur caused by rotating speed fluctuation. Then, the angle-domain signal was processed by signal synchronous average, weakening the frequency components of being irrelevant to speed, improving the signal to noise ratio. Finally, the averaged signal was demodulated, extracting the main meshing order components of the gear fault from the automated edited order spectrum and recovering the angle-domain signal. Then it was performed by amplitude and phase demodulation which was based on the frequency shift. The fault features of gear can be extracted in accordance with the demodulation waveform diagrams. Simulations and tests were conducted respectively to verify the validity of the proposed method.