In production process of tandem cold rolling, abnormal vibration of rolling mill may cause product quality problems and even strip breakage to greatly limit production efficiency. At present, online flutter monitoring is realized in iron and steel enterprises, but the monitoring system can only suppress vibration through vibration alarm and then speed down. Here, aiming at the difficulty to establish accurate traditional mechanism model for cold rolling vibration, using production data measured on site, the gradient boosted decision tree model was established to perform vibration energy regression. Characteristics of the gradient boosted decision tree algorithm were used to do feature selection, find important factors affecting vibration and simplify the model. The simulation results of actual production data showed that the model established with the gradient boosted decision tree can effectively select important factors, and reduce its complexity; the established regression model can accurately reflect the varying trend of rolling vibration energy.