For the chaotic motion of a single-stage gear system with tooth backlash and bearing clearance in some parameter region, the unstable periodic orbit of chaotic attractor was stabilized by improved OGY chaotic control principle. The OGY control for a smooth dynamic system was transformed to a non-smooth multi-dimensional dynamic system by the finite difference method instead of the Jacobi matrix under non-smooth point. And PNF method is used to search the pseudo-fixed points of unstable periodic in chaotic region. The Jacobi matrix and sensitivity column vectors in the improved OGY algorithm were solved according to the dynamic equation and variation form, the periodic orbital interval and transfer characteristics of chaotic attractor were analyzed combined with Poincaré mapping. The simulation results show that the improved OGY control method is also effective for chaos control of multi-dimensional non-smooth gear systems. When the multi-period orbit was continuously controlled, the difficulty of chaos control increases with the increase of the target periodic orbit, and the required parameter perturbation increases correspondingly.