To improve isolation performance, characteristics of force transmissibility of the vibration isolation system with geometric hysteretic damping , which raised by friction damping under nonlinear geometric condition, is investigated. Different from traditional single-degree-of freedom vibration isolation model, a unconstrained two-degree-of-freedom vibration model is built, which included the influence of the mass of the basement. The motion equation is solved by using the averaging method to get the frequency-amplitude equation. Then the expression of the force transmissibility and the force-displacement transmissibility is derived. It shows that mass, damping and excitation frequency are the dominant parameters influence the transmissibility. Comparing to linear viscous damping, the considered damping has better performance in vibration isolation, not only in suppressing the resonance, but also in improving the isolation performances at high frequencies. At last, a numerical solution of the motion equation with sweep-frequency force excitation is calculated and compared with the theory results. It turns out that the two results match well, and the geometric nonlinear damping can lead better performances to vibration isolation systems.