A time-frequently approach based on Hilbert vibration decomposition method is introduced in order to extract fault features of multi-component mechanical fault vibration signals accurately. Firstly, the analysis signal of original vibration signal was obtained through Hilbert transform. Secondly, the non-stationary frequency of the largest component was achieved as a low-pass filter of the instantaneous frequency, the corresponding envelope and initial phase was also estimated according to the synchronous demodulation, then the time-frequency information of each component of the initial signal would be detected by each iteration step. Aiming at overcoming the end effects of HVD, a wave characteristics matching extending method based on correlation coefficient criteria is proposed to improve HVD. The analysis of two simulated signals showed a good capacity of HVD in decomposing the non-stationary multi-component signals, and the results inflected that the improved HVD inhibited the end effects. Finally, an oil whirl of rotor system fault diagnosis instance is given to validate the feasibility of this method.