In order to catch typical fault characteristics of steam turbine generator set rotor under nonlinear and nonstable instance, a new instantaneous energy (IE) distribution characteristics extraction method based on Hilbert-Huang transformation (HHT), which is used to recognize different operating state (such as normal state or oil film turbulence of rotor system. Analyzed physical relationship in theory between instantaneous energy and structural state variation of system, by the use of HHT, obtained instantaneous energy distribution feature of unstable vibration signal, and eliminated noise according to relative contribution quantization -based method, discussed the typical character of instantaneous energy using correlative coefficient, presented a new fault diagnosis method for steam turbine generator set rotor based on instantaneous energy distribution feature, and the results of experiment proves that this kind of method is effective.