It is difficult for controlling and predicting the motion law, the dynamics action and the reliability of the drilling mechanism in drilling coal rock process, which are due to the complicacy of coal rock, the contact impact between drill pipe and coal rock, the coupling vibration of the Coal rock –Drill bit –Drill pipe system. Based on this, considering the drilling load of drill bit and coal rock, contacting load of the drill pipe and coal rock wall, and dispersing the drill pipe into a number of finite elements, the coupling nonlinear dynamics model of the drilling system is built. And the vibration coupled the longitudinal, transverse and torsional is studied under the conditions of different coal rock hardness and different rotate speed of drill bit. And discuss the vibration of drilling mechanism through experiment under different conditions. The results indicate that the transverse vibration radius, the longitudinal vibration frequency and amplitude are all reduced when the coal rock hardness decreases; and for the same coal rock, the biggest transverse vibration radius has exponent relation to the change of drilling depth; the effect of rotary speed change on the transverse vibration is small, which only changes the amplitude of the longitudinal and torsional vibration; and the transverse vibration and longitudinal vibration are generated by taking the torsional vibration as the exciting force, Which provides a theoretical basis for the design of drilling mechanism.