In order to consider the influence of nonlinear characteristics of the joint surface on the dynamic characteristics of whole machine, the composite beam is taken as a research object, and the nonlinear pre-stressed modal analysis method is provided based on ANSYS. The nonlinear characteristics of the joint surface are proved to have an important influence on the natural frequency of the composite beam. Moreover, the influence of normal load and friction coefficient on the natural frequency is revealed with single variable method. The results show that the natural frequency is decreased by normal load and friction coefficient, and friction coefficient is the dominant. Finally, the modal test is conducted and the mode shape of composite beam got by the modal test and pre-stressed modal analysis is coincided. The natural frequency error between them is in an acceptable range which demonstrates pre-stressed modal analysis is correct.