Oil whirl of steam turbine is a self-excited vibration, which results from instability of sliding bearing. The vibration frequency of oil whirl is denoted by half of the rotational frequency of rotor shaft or less. Once the frequency of oil whirl is equal to the first critical rotational frequency of rotor shaft, the vibration of steam turbine is intensified and the stability of steam turbine can be destroyed. Gabor transform is a reversible joint time-frequency distribution method, inverse transform of which has the ability of time reconstruction. In this paper, the vibration signals of 850 MW steam turbine are analyzed based on Gabor transform, and the semi-speed oil whirl component representing instability is detected, then the semi-speed component is separated using band-pass filter in time-frequency domain and reconstructed using inverse Gabor transform, at last a peak-to-peak quantization index is calculated to judge the level of oil whirl. The presented method can provide evidence for oil whirl fault diagnosis of steam turbine.