Combined with the Chebyshev series theory and nonlinear optimization algorithm, a method is proposed to calculate the periodic solution of nonlinear vibration systems. The method transforms solving the unknown Chebyshev coefficients of the state vector into the optimization problem on solving the minimum value of residual over one primary period, and a high precision Chebyshev series periodic solution is obtained. The Floquet transition matrix can be calculated by periodic-solution integral operation. Then the stability of periodic solution can be analyzed. At last, two examples, namely Duffing equation and helicopter rotor motion equations, are proposed to demonstrate that the method is correct and effective. The examples also prove that introducing the Chebyshev series theory into the helicopter aeroelastic response and stability study is accuracy and feasible.