This paper aims at the shortcomings of the existing smoothed particle hydrodynamics(SPH) methods in the dynamic meshing and transmission process analysis of gear rack and other mechanisms with complex boundaries, and improves the gear rack meshing and transmission characteristics.First, by improving its interaction force, an SPH calculation method for dynamic meshing and transmission analysis of complex boundaries was proposed, and the corresponding SPH program was programmed.Secondly, a simplified gear rack meshing and transmission example was analyzed by using the SPH algorithm, and the information of gear rack position, velocity change, and stress distribution at different times were obtained.The mechanical principle of tooth surface pitting and root failure of gear rack mechanism was revealed, and the feasibility of the method was verified by comparison with a finite element method to verify the feasibility of the method.The proposed SPH method in this paper can be widely used for strength evaluation and optimization analysis of mechanisms with complex dynamic contact boundaries such as gear, rack, and bearing.It provides a new idea for numerical analysis in this field and also promotes further development of SPH in this field.