Rigid rotor system supported by cylindrical roller bearing displays complicated nonlinear vibration behavior due to nonlinear Hertzian contact force. The nonlinear force and moment caused by roller bearing are calculated based on relationship of displacement and deflection, and four-DOF dynamic equations of rotor roller bearing system are established. Newmark-β method and Newton-Raphson method are used to solve the nonlinear equations. The dynamic characteristics of roller bearings are computed considering the nonlinear vibration of the rotor system, effects of vibration of rotor system on bearing dynamic characteristics are analyzed, and dynamic performance of roller bearing are tested in the test bench. The results show that the slide ratio of cage, stiffness of roller bearing vary obviously at non-periodic vibration range of rotor, which may cause failure of bearing. Prediction accuracy of the dynamic characteristics of roller bearing has been greatly improved considering nonlinear vibration of rotor system.