The drilling rig's driving head transmission system is studied based on the method of gear system dynamics. The contact force of cutting rock by PDC bit is studied based on the theory of elastic mechanics and contact mechanics, and time-varying input torque of gear transmission system of drilling rig's driving head, caused by Verying Working Environment is then obtained. A coupling dynamic model of the gear transmission system is built using lumped-parameter method, the model includes the time-vary mesh stiffness of gear pair, the stiffness of rolling element bearing, the meshing error and etc. Then dynamic differential equation is built on that basis, and vibration responses, dynamic meshing forces of gears as well as bearing forces of the gear transmission system are obtained using numerical method. Research results lay a foundation for dynamic performance optimization and reliability design of the gear transmission system of drilling rig's driving head.