Planetary gearboxes are widely used in various types of machinery. Fault diagnosis of planetary gearboxes is a very important topic. A method based on Hilbert vibration decomposition and higher order differential energy operator is proposed for planetary gearbox fault diagnosis. By exploiting the advantages of low computational complexity and mono-component decomposition ability, Hilbert vibration decomposition is used to decompose a signal, thus fulfilling the mono-component requirement by higher order differential energy operator. Considering the merits of fine time resolution and good adaptability to instantaneous changes in signals, higher order differential energy operator is applied to detect the transient impulses induced by gear fault and to estimate the corresponding amplitude envelope and instantaneous frequency. Fourier transform is applied to the higher order differential energy operator output and the obtained amplitude envelope and instantaneous frequency, to identify the gear fault characteristic frequency and thereby diagnose planetary gearbox fault. The effectiveness of the proposed method is validated with both numerical simulated and lab experimental signals of a planetary gearbox.