The dynamic behavior of the rotating composite thin-walled shafts with geometrical nonlineary is studied in the paper. The nonlinear extensional-bending-torsional equations of motion for the rotating composite thin-walled shaft are derived using Hamilton’s energy principle and variational-asymptotical method (VAM). On the basis of von Karman’s assumption, the geometrical nonlineary is included in the relationship of strain and displacement of the shaft. In order to emphatically study nonlinear transverse bending vibration, the effects of extensional and torsional deformations are ignored.Thus, the nonlinear transverse bending equations of motion for the rotating composite thin-walled shaft are obtained, in which external and internal viscous dampings are also considered. Galerkin’s method is used to discretize the governing equations and the ordinary differential equations of the rotating shaft are obtained. By using fourth-order Runge-Kutta method the time respone waveforms, phase plane curves and power spectrums are obtained. The study shows the effect of the external damping, internal damping, eccentricity and rotating speed on nonlinear dynamic behavior of the shaft. Specifically, the numerical simulation results shows that the shaft may exhibit chaotic motion.