The vibration control of Active Magnetic Bearing (AMB) flexible rotor system was studied. Five degrees of freedom mathematic model of AMB rotor system was established firstly. Method for selecting weighting functions based on the vibration amplitude of the rotor in motion on natural frequencies was put forward. The H controller of the system was designed according to the mixed-sensitivity method. The simulation analysis and the experiments of static suspension and high rotation speed of the system were performed to validate the influence of the H controller on dynamic characteristics of the system. The results show that, the H controller can help to reduce the vibration amplitude of the rotor running in motion on natural frequencies and the system is robust.