In the process of rock-crushing on bottom of hole， drilling pressure、self dynamic loads、random impacts and contacts of cone bit and rock on the roller cone bit system is considered. With alternate pressing-in， scraping and rolling， the motion state and stress of cone bit is extraordinary complex. Based on plastoelastic constitutive equations of rock and equivalent plastic strain damage principle also considering the confining pressure and random loads of bit and rock， nonlinear dynamic model of roller cone bit-bottom rock system is set up using the FEM method. Kinematics law and mechanical property of roller cone bit system are discussed. Also the formation process of craters on the rock is stated in this paper. The research results lay down a theoretical basis to forecast the formation of hole trajectory、establish foundation for nonlinear dynamic theory of drill string system coupling vibration in the actual work condition.