As two main obstacles of rolling bearing diagnosis, variable rotational speed and gear vibration noise, although they can be partly solved by order tracking based on rotational speed information from tachometer and ANC algorithm using another reference sensor, the rotational speed information and reference signal is not easy to obtain in real engineering due to the design or cost reasons. To solve this problem, a new algorithm with five main steps is proposed in this paper: a) a revised ANC algorithm using newly constructed reference signal based on instantaneous dominant meshing harmonics (IDMH) trend can weaken the interruption from the gear vibration of identifying the resonance frequency from faulty rolling bearing, b) fast algorithm of spectral kurtosis (SK) method can be used to determine the center frequency, the frequency band and corresponding scale with which the filtered envelope signal which displays the rolling bearing fault extremely can be extracted, c) calculating the envelop time-frequency representation of two-filtering outcome from which the instantaneous fault characteristic frequency (IFCF) can be extracted using the peak-searching algorithm, d) proposing a new resampled algorithm based on sampling rate resetting and resample the filtered outcome into fault angle domain, e) identifying the operating state of rolling bearing using the fault characteristic order (FCO) spectrum from the resampled signal using FFT. The effectiveness of the proposed method has been validated by both simulated and experimental bearing vibration signals.