For the integrated relative displacement self-sensing magnetorheological damper (IRDSMRD) with a constrained volume, a Pareto optimization principle based method, which optimizes the key parameters of the IRDSMRD taking the linearity of the integrated relative displacement sensor (IRDS) and the damping force of the IRDSMRD as the objective functions, is proposed and realized in this paper. After modeling and analyzing the magnetic circuit property, which influences the damping force and the linearity of the IRDS, the mathematical models of the linearity of the IRDS and the damping force of the IRDSMRD are established. The Pareto front, representing the best tradeoff between the damping force and the linearity of the IRDS of the IRDSMRD, is obtained by considering a volume-constrained IRDSMRD. The research results indicate that every point on the Pareto front is an optimal solution that compromises the damping force and the linearity of the IRDS of the IRDSMRD, and the optimal damping force of the IRDSMRD with a certain linearity of the IRDS as well as the optimal linearity of the IRDS with a certain damping force can be determined according to the Pareto front for a specific application.